xorcise your daemons

ramblings on software development

June 25th, 2009

Why is engineering software so hard/different than other engineering pursuits ?

I’ve been pondering this topic a lot lately, and have had several discussions with
colleagues and friends on what steps one might take to improve matters when working
on a project which has issues associated with this topic.

This Dr. Dobbs article brought the topic to
the forefront of my mind and has recently resulted in the following thoughts on the subject.

1) The solution to any given software problem has far more possible solutions than in
other fields of engineering, and tends to be on the bleeding edge of technology. Other engineering fields
don’t tend to put developments out of their R&D labs into customer usable/facing products
nearly as rapidly as in the software industry. Effectively, customers are most commonly using the bleeding
edge instead of a product from tried and true methods, processes and technologies. This
is a fairly sizable contributing factor in the complexity of the creation of software. The fact
that developers are learning new technologies and techniques all the time and applying them
each day is a huge testament to the traditionally less than strict process involved in our industry.

2) The group members in any engineering team are of different quality by any metrics you might measure them by.
How that impacts a project and subsequently the whole team, is where things diverge when comparing software development
to other engineering disciplines. Only in the software industry can one person have as large an impact on the whole ecosystem.
If you’re building a bridge, and one contributor isn’t keeping up, you send them packing and find another in short order. In the
software realm, you could send the person packing, but usually don’t, and whether you do or not there’s a much larger impact.
Replacing someone usually doesn’t happen when it needs to, takes far too long, and is far too difficult to do successfully.
This needs to be dealt with case by case, as every company and environment will be able to address it different ways.
The goals should be to a) have processes in place that make it trivial for people to switch projects on their own, including
capturing their knowledge on current projects. b) allow the team to decide how best to divide the work it’s been given c) force
the team to take responsibility for itself and its project. d) implement current standard practices such as bug tracking and
continuous integration. You might be surprised how regularly such things are either not in place, or not used consistently…
They can help bridge the skill gap to a certain degree by allowing the team to understand which of their colleagues is capable
of what, and can allow them to compensate internally without any intervention. In the software industry
the least capable person has a far greater ability to affect the project than in other industries. Addressing this single
point would go a long long way toward improving the software engineering domain. Sadly, I have no idea what
to suggest, and every attempt I’ve seen to do so has been a failure or marginal success.

3) A view point offered by my manager, Eric Kihn,
is that there’s such a large uncertainty associated with estimating both new features and bug fix efforts, that it’s difficult to plan a software
project to any reasonable degree of certainty when compared to other engineering efforts. Again, if you’re building a bridge it’s far easier
to know if you’re on schedule, and if not what your trouble areas are and how to address them. Too commonly a software developer
will jump into a feature or bug fix and find out that what they thought would take a week will actually be more on the order of months.
Experienced software engineers are obviously better at estimating, but the potential impact of the unknown is still far larger than in
other engineering fields.

4) As the Dr. Dobbs article points out, the human factor is a massive contributor to this phenomenon. I’ll expand
that to point out that because of the human factor, architecture religion/politics/preconceptions all impact a project
way too much, and way more than in any other industry. At least in terms of getting things done. I’m sure bridge
builders have politics too, but it seems to be less about the methods used than the process of doing the work in that case.
If I’m a commuter who’s going to drive over a bridge, I don’t care what brand of tool was used to lay down the asphalt, I
just want the bridge to work reliably day after day. If I buy a house, I don’t care what brand or model of drill was used
to screw in my screws, I just want the house not to fall down, and last year after year. In the software industry we
spend way too much time arguing over what brand of drill we’ll use to create a piece of software. Pick technologies
that make sense for the project, the environment in which it must work, the constraints of the company, and the experience
of the team doing the work. In other words, stop arguing about ultimately pointless things and make sound decisions based on
a project’s goals and constraints.

The goals of a project will determine what makes sense. If the right people understand this and provide
proper guidance, software development can be more like other more stringent engineering pursuits. What
value would we or the product get from that though, and does it matter ? Who knows, but over the course
of my career I’ve heard one question far more than any other. “Why is it so hard to write good software ?”

Popularity: 5% [?]

June 12th, 2009

The Tipping Point

I added The Tipping Point to the books page.

This book was recommended to me by my former mentor at Sun, and
was quite good. Follow the link or go to the books page for more details.

Popularity: 41% [?]

May 30th, 2009

jpeekaboo updated, new features added

Something finally motivated me enough to make some changes to jpeekaboo.
Time does indeed fly, it’s been almost a year since the last release…

I put in some infrastructure for settings, and used that to allow the note
to be pinned to either the left or right side of the screen. The settings
also have a way to set the vertical percentage of the screen the note will
occupy now as well.

Grab yours here

Popularity: 43% [?]

May 29th, 2009

First wolf3d, now Doom for iPhone!

John Carmack commented on the existence and forthcoming release of Doom for the iPhone.

That will certainly be interesting, and per the article more time has been put into the interface (among other things) so it should be a better app than wolf.

Hopefully he’ll release the Doom iPhone source code too…

Popularity: 23% [?]

May 27th, 2009

now that’s a great idea! (data.gov)

The government decided to do something useful with a few of its copious data stores, excellent!

After perusing data.gov a bit, there are quite a few useful data sources. More importantly, this
could be much larger than it currently is, and represents a huge opportunity for so many federal organizations to share their data
not only with one another, but with us normal folks too.

Imagine having this sort of programmatic access to a wider range of data over a wider time frame.
It puts all sorts of interesting statistical and scientific measurement opportunities into the hands of a
great many people.

It will be very interesting to see what comes of this effort.

Popularity: 41% [?]

April 6th, 2009

MapReduce becoming ubiquitous ? => Collaborative Map-Reduce in the Browser

As an academic exercise, myself and a colleague at Sun are currently implementing a MapReduce client for iPhone which has a server component, and a number of other clients we’ve written in various programming languages (currently Python, Ruby and Perl). Our hope is to statistically prove iPhone and similar devices can be useful in a larger MapReduce exercise to augment beefier hardware in the data center,
specifically for processing massive amounts of textual information for the purpose of search and data mining.

Assuming you have enough mobile device toting participants of course…

In the course of our implementation we’ve come across a few other interesting ideas in this
space, such as this one which creates a MapReduce client in Javascript to farm compute to website visitors, which is very neat.

Also, Amazon recently announced elastic MapReduce.

More and more lately I’ve been sensing that MapReduce is seeping into software developer’s common knowledge, which is a good thing.

There is a class of problem for which it’s extremely applicable, but like anything, it is NOT a silver bullet, nor
the solution to every massive scale data processing problem. Alas, as is too common, it seems to be the next big shiny thing…

After its completion, I’ll provide additional details and possibly some source code on our iPhone MapReduce project.

Popularity: 50% [?]

April 1st, 2009

arrrrrrrg, thanks a bunch site5… irudite.com photo prints

A week or so ago I created a photo print site/store to post some of my photos
and possibly sell some high quality prints. To do this, I had to get a static IP and
SSL cert from my host, site5.

Well, I have a number of domains parked with site5 to direct things like elespuru.com, peterelespuru.com
and such to this blog, and site5 decided to leave some of the parked domains out when updating
DNS for the new IP, sigh.

If you got directed to the generic Apache/cPanel page when hitting one of those sites, sorry…
Let this be a lesson to anyone with shared hosting, DOUBLE CHECK THEIR WORK!

Anyhow, here’s my new photo print site, which is just beginning. I scoured many thousands
of photos I’ve taken over the years, and posted ones I felt might work as prints. As I get more
serious about this as a hobby with the new camera
I’ll continue adding content.

Popularity: 49% [?]

March 25th, 2009

wolf3d on iPhone!

An immortal classic on iPhone at last, Wolfenstein3D!

Buy it, or, grab the source and compile yourself!

Many thanks to John Carmack for putting the source code out too, and thanks to Josh Buhler for saving me the trouble of resolving the compile errors myself ;)

per his post, Josh doesn’t have a cert so couldn’t deploy to a device, and as such didn’t run into the certificate issues. So, I’ll amend his post to add those steps which are necessary to get it all the way to a phone/iTouch.

  • Under the ‘Project’ menu of XCode choose ‘Edit Active Target “wolf3d”‘
  • Choose the ‘build’ tab
  • Choose a configuration (Debug, Release, whichever you’re deploying with)
  • In ‘Code Signing’, change to your cert
  • Be sure to update ‘Bundle name’ in Info.plist (see here)
  • Clean all targets, and build
  • That’s it, it should deploy now

Kudos to devs who put out source code, the more to peruse the more to learn from…

Popularity: 64% [?]

March 10th, 2009

Making Things Happen

Added Making Things Happen to the book page.

Check it out.

Popularity: 50% [?]

February 20th, 2009

programming language typing, static and dynamic

I just read Bruce Eckel’s blog entry which discusses programming language typing, static vs. dynamic.

His conclusion is that we, software developers, need “Strong testing, not strong typing.” and I couldn’t agree more. I’ll expand on his statements and reiterate that all too often software folks get caught up evangelizing their chosen technology and lose sight of their end goal.

Take a step back. Ask yourself WHY you’re doing something. Ask yourself why more often. Ask your peers/colleagues/friends for their opinion and explain to THEM why you’re making the choices you are. Now comes the hard part, actually listen to their feedback.

Bruce’s example of Python is quite good and I applaud his explanation and level of detail. Having an in depth example from the person that wrote “Thinking in Java” is a great conversation piece and point of reference for those of us who are more concerned with solving problems and getting things done than evangelizing technologies. In an industry where the only constant is change, we must get better at picking the right tool for the job.

Popularity: 54% [?]

February 18th, 2009

root cause analysis - keeping software simple and high quality

I posted a brief thought on RCA on my Sun blog

In short, software developers could improve quality and simplify by employing RCA during development. Avoid getting caught up in what’s new and cool and focus on what your intended users really need.

Popularity: 56% [?]

January 17th, 2009

XCode breakpoint speech synthesis!

In mid December I attended iPhone dev camp Colorado, and among many other things, learned about the very cool ability of XCode to speak the value of variables and other items at breakpoints while debugging.

XCode uses whatever voice you have configured in OS X to do the synthesis, and spits out not just variable values and the like, but additional verbiage to explain what it has encountered.

Think about it. You’re debugging an application and want to focus on code while you’re stepping through. Just enable speech for the break point and there’s no need to take your eyes off the code!

Combined with a premium voice from Cepstral and you have an extremely handy and gee whiz debugger on your hands.

Give it a go, by reading up on break point templates here

Popularity: 51% [?]

January 7th, 2009

Rating Software Developers

It takes all kinds of people to make any project successful; designers, code monkeys, requirements
junkies, architects, and there are no repeatably good mechanisms for evaluating folks to determine
if you want them to be a part of your team or not. This sort of rating or measurement is something people from
all three perspectives really need to make a good decision. Hiring managers need to know they’re
getting a good employee, employees need to know where they stand, and if/how they’ll contribute, and
their peers need a way of determining if they can work with the person.

If you only focus on a subset of a typical team, say the coders, how can you really evaluate
them to decide if they’re good or bad, or maybe just unsuited for your team ? How can they
evaluate themselves to decide if they even want to be a part of your team ? (which
is just as important!)

To date I’ve seen a number of approaches to hiring and evaluating software people, but I won’t
go into those details, as they’re not the point of this rant. The point is the people doing the hiring
ask such a variety of questions and seem to focus on trivia or things Google can answer. They look for experience with specific programming languages,
frameworks, operating systems, and formal education; none of these things tell you if you’re looking at a
prospective employee who can actually get anything done. Too little weight is given to people who’ve proven
they can learn. Too typically, expertise in a small set of technologies wins out over other factors, which is
a shame, after all, the only constant in this industry is change.

Software people are measured by their experience, education, possibly
some certifications from brain bench and their ilk and perhaps
even some discussions regarding how many lines of code they wrote
for project X in their past, and how amazing it was.

Take a look around you, really look at your peers; do any of those
things correlate in any way to their actual contribution to the team ?

This is a rather large unsolved problem in my opinion, and gives me a
whole new level of respect for recruiters who do their job well.

Perhaps there’s some formula that would more accurately
describe a programmer if applied to enough of their work over time.

If for example on each project they’re a part of, they kept track along the lines of…

(time to complete task * task complexity factor) + result quality + task’s percentage of project = task rating

If you average every developer’s task contribution over enough projects this way, would it help ?

Is this valid ? Is it an improvement over what’s available already ?
I don’t know… What I do know is that the current ways need improvement…

Popularity: 49% [?]

January 3rd, 2009

iPhone development joy

Seeing as it’s been a year now since I began coding for iPhone, I felt like rambling a bit on that topic…

I got my 1st gen iPhone in September of 2007, and knew within minutes of actually using the thing (rather than just drooling over photos and videos) that iPhone was going to be huge. The end user experience is too amazing for that not to be the case.
I upgraded to a 3G as soon as I could, which happened rather uneventfully (no lines, no wait) in July of 2008. Initially, as it did for most, my battery life sucked. Fortunately, Apple released a firmware update, that for me at least, made that pain a issue of the past. It’s been smooth sailing ever since, and as anyone who knows me will attest, separating me from my phone would be more difficult than moving a mountain. One colleague even jokingly asked me what I would do if I had to choose between my iPhone and my wife…

My first application for iPhone was a call forwarding toggle that I implemented last January, using the GCC tool chain and my jailbroken 1st gen iPhone. The problem I had is that the built-in mechanism for enabling/disabling call forwarding is 5 or so levels deep in the system configuration app, which is a rather lengthy pain if you want to quickly change your forwarding settings. I have an Asterisk based PBX I run on a server at home that provides phone and fax to our house, and every evening when I return from work I have my cell forward to the house system. When I leave in the morning, I disable forwarding and allow the cell to do its own work during the day. This forwarding application allowed me to quickly turn forwarding on/off as necessary, just by launching the application.

The system settings application provided by Apple uses private APIs to interact with the carrier network to deal with these settings, but it is possible to dial GSM codes via dial URL, or manually from the dial pad to accomplish the same goals, which is what my app used. That is, until the official SDK came out and disallowed the use of certain special characters in a dial URL… Specifically, the special characters necessary for GSM codes, despite the fact that you can still dial them on the keypad.

Sigh.

When the official SDK came out, my first app was a collaboration with a colleague to spruce up the forwarding app and allow multiple numbers to be stored, so one could have multiple forwards at the ready. Sadly, we didn’t find out about the character limitation until everything else had been coded… The issue is that an attempt to open a URL with a dial string which has special characters silently fails. No errors, no warnings, no nothing. We used one of our two developer account support requests to ask why, and were told by Apple that it isn’t supported by the public API, and that if enough requests are received for opening dial characters up it may change some day. So, App #1 was shelved and work began on several other applications in our free time and solely for fun. So far two have made it to the AppStore, and while they’ve not made anybody rich, they’ve more than paid for the cost of the developer licenses, which is just fine. At the moment, there are 5 others at various stages and I continue to struggle to set aside enough time to get anything meaningful done on them these days. They’re all more ambitious than the current two, and use more aspects of the device, which makes development a lot more involved.

All in all, iPhone development is very rewarding, and Apple has done a phenomenal job of providing clear high level abstractions for dealing with complicated device internals. It requires very little effort to use data from all of the most interesting bits of the device, such as accelerometers, touch events and location services. I’ve done a fair bit of user facing development in my day, and can honestly say no other SDK/API/etc comes close. The iPhone SDK/APIs do exactly what they should, they get out of your way and allow you to make interesting software.

It would be great if Apple would open up the private APIs for video, phone etc. It’s a bit perplexing that they’ve held some stuff back, and makes one wonder who/how they decided what is/isn’t “private”.

Kudos to Apple for putting together a hell of an end-to-end environment. Nothing is perfect, but they’ve done a better job of bringing it all together than anything else I’ve ever encountered, with the IDE, simulator, device, SDK/APIs, marketplace and community, all that’s left is for creative people to cook up the goods.

Popularity: 65% [?]

December 14th, 2008

soekris embedded linux devices

For years I’ve had a number of systems that have been retired to
my server closet as they’ve lost favor as my primary desktop, only to be reborn as
a server of this, that, or, something else…

Unfortunately, these systems are power hogs and are getting old enough that replacement
parts are available only from a museum.

A few months back I decided to remedy this situation and started by replacing my
VPN server (OpenVPN) with a nifty embedded
linux box from soekris.

The soekris are a bit pricey, compared to say a WRT54G using OpenWRT, but I gave them
a go anyway and have been thoroughly pleased. So pleased in fact, that I’m in the process
of replacing my home web/db server that I use for a number of personal tools and toying with ideas,
my firewall box, and another server that powers the house PBX system (Asterisk). The current
web/db server is a Pentium 3 system circa 2001 or so. The PBX is slightly newer, but is an eBay special
Dell small form factor desktop that was probably new sometime in 2002. The firewall system is
another eBay special, but is an HP small form factor desktop that likely began life in 2003 or so.

Less power consumption, more efficient use of what hardware is available, and no more systems
that have ISA expansion slots.

After this project, and some additional consolidation, the only non-soekris will be the house wide
NAS which is a 4U rack mount with 3.5″ 7200 RPM SATA disks driven by OpenSolaris and the ZFS
file system.

check soekris out if you need some reasonably priced
embedded boxen. They have a wide array of configurations.

Popularity: 69% [?]

December 11th, 2008

JavaFX

another day, another scripting language…

I’ll reserve in depth comment until I’ve had time to thoroughly explore JavaFX, but the choices
facing today’s software folks are staggering to be certain. Personally, I’m finding it harder
and harder to keep anything in my own memory, syntactic details and the like. More often
than not these days, google serves as my memory, sigh.

maybe I should create some flash cards for all of the programming languages I work
with regularly… or, stop context switching among so many languages, but then where
would I be ?

Popularity: 65% [?]

December 11th, 2008

Unified Storage Simulator and the 7000 series NAS appliances

Have a NAS ?

Have a NAS that can tell you graphically which client(s) are hogging the NAS while pushing/pulling which file(s) ?

yeah… I didn’t think so.

November 10, 2008: Sun announced the 7000 series storage products, which combine some very useful administrative features with the power of Solaris and ZFS.

I’ve had the opportunity to play with these and have been thoroughly impressed in so many ways. For businesses ranging from small to large, the minuscule time it takes to get up and running just begins the journey of pain free storage systems. For even smaller shops, OpenSolaris allows you do to largely the same thing, but requires a lot more effort and time to set up manually to accomplish the same goals, and lacks the graphical analytic capabilities and simple appliance style management, but can be done for free if you’re strapped for cash.

Don’t take my word for it, Get the FREE VMWare virtual machine and give it a try your self!

Yes yes, cheer leading for my employer, but I truly feel these products rock, as do OpenSolaris and ZFS. Again, try it yourself, compare to what
you’re currently using, and come to your own conclusion…

Popularity: 70% [?]

November 5th, 2008

gRApple updated to reflect new Apple refurbished store layout

Apple updated their refurbished Mac store and in the process broke gRApple.

So, I updated gRApple to reflect the Apple site changes, it’s available here

Apple actually improved the parse-ability of the refurb store this time around so the changes were
pretty simple and involved removing some older extra bits in the code as well as making a few search element
changes to align with the new document layout.

Anyhow, get your refurb filtering and purchasing on, lots of excellent deals to be had at the moment!

Popularity: 52% [?]

October 23rd, 2008

mmmmmmm, ZFS.

First, the background info… We have a family wide NAS which houses hundreds of movies, tens of thousands of songs, hundreds of documents and all manner of other important digital content like vacation photos.

Recently, I was listening to some old music (~5yrs) and noticed some degradation (very audible pops and clicks), which is unacceptable. The NAS was running Debian and a software RAID5 (4 disks, 3 live, 1 hot spare), and had been in that form for a few years. Now, the corrupt audio could easily have preceded this particular incarnation of the NAS (which has been around in various forms for ~12yrs), but it was concerning nonetheless.

I’ve read and seen lots of phenomenal press regarding ZFS and decided this was the time to give it a go. So, I ordered some external drives from Newegg to use as a staging area, copied all the existing data onto them, loaded the NAS server up with Solaris, and created a mirrored ZFS pool from all 4 disks (I’m more concerned about integrity than overall size at this point).

Solaris supports the two primary protocols the heterogeneous NAS clients connect from, CIFS and NFS, and I have a weekly ZFS scrub running in cron. To top it off, data I/O is noticeably faster on Solaris than it was Debian.

All in all, I’m thrilled. Solaris could be less painful, but ZFS is an amazing file system that lends me confidence regarding data integrity of my many irreplaceable files.

If you haven’t played with ZFS yet, you’re missing out. Grab OpenSolaris, and explore.

You can even create ZFS file systems on top of files (instead of disks) if you want to try additional things out, but lack additional disks! Try that with a MD software RAID…

Popularity: 59% [?]

September 30th, 2008

Android SDK

I’ve been developing for iPhone in my spare time for ~8 months now (jailbreak and toolchain prior to the official SDK). For the sake of trying out the competition I snagged the Android SDK and associated plugins and whatnots for Eclipse, and proceeded to implement a simple app.

My 2c is that Apple wins on most fronts (ease of use, abstraction, API, framework elegance, etc…), but Android development was straight forward enough and I do see the potential. Hopefully the competition will do the mobile platform front some good for all parties…

Android development has a long way to go, and the SDK and toolkit feels fairly hacked together IMO. I’ll probably develop for Android too, but plan to wait and see how the Android Market and G1 uptake develop before diving in completely. It just doesn’t feel as “huge” as iPhone did from the outset, so isn’t really pulling me in to the same extent.

Popularity: 76% [?]

September 9th, 2008

Chandler 1.0

Most folks may not be familiar with the Chandler project, but it has some interesting goals. The project finally
hit 1.0 recently, and reached a point that it’s stable enough for serious usage, IMO. I’ve used at various points in the past, and I must
say that they’ve done a great job. I’ve been using it since 1.0 went live a while back, and have been very pleased.

Until now, I’ve been using the auto hide drawer note of my own creation to handle to-dos and general
note taking. Chandler stays out of your way and, for my work flow tastes anyhow, does a great job capturing things as they come up, while also dealing with longer term notes and task items. Its syncing abilities and web interface make it superior to any similar app, and I recommend it for anyone that uses GTD like apps, or other methods for keeping track of their tasks etc…

I started reading Dreaming In Code, which chronicles Chandler’s beginnings, but I must admit I found it too boring to finish. I only managed to finish 2/3 before losing interest. It is an interesting read from a historical perspective, and perhaps as an example of how other organizations develop software, but for my tastes it just couldn’t keep me engaged.

Anyhow, if you haven’t heard of it before, or haven’t checked it out, go now!

Popularity: 76% [?]

July 24th, 2008

jpeekaboo v1 released

Alrighty, I finally put the finishing touches on jpeekaboo tonight, enough for others to try it anyhow.
v1 is out, and is essentially at the same place as where I had my Stickies fork, only without
all the extra unused stuff, and in java!

native distros for linux, OS X and Windows are available from google code

and here’s its dedicated site:
http://jpeekaboo.xorcyst.com

on a related note, my foray into C#/.NET was interesting indeed, but I’m too big on having cross platform capable apps for that to ever agree with me,
and no, I don’t want to mess with Mono. So long C# and thanks for all the, er, fish ?

Popularity: 82% [?]

July 11th, 2008

jpeekaboo, java based auto hide drawer note

pwsidenote is becoming far more painful than I’d hoped to strip down and end up with what I want.

In response, I’m shelving it as is and starting a new project, jpeekaboo.
Same goal, auto hide drawer note that can be pinned to any screen edge and sync content offsite.

I also need an app that can run on linux, OS X and windows, so that’s a fair portion of the
motivation too.

I’m sorting through the prototype, but it’s coming along nicely. I’ll post the details when it’s ready.
In the mean time pwsidenote will stick around, but won’t be developed anymore.

Popularity: 93% [?]

June 26th, 2008

pwsidenote, an auto hide drawer style sticky note application

I’m always on the lookout for small apps that help me improve my efficiency. Despite many wanderings into the
land of GTD apps, I always come back to a simple plain text scheme, using either post-it like applications, or a simple
text editor. On the OS X platform, I use sidenote. On Windows, I use Stickies for Windows.

I recently found myself wishing Stickies was more like sidenote. Fortunately, Stickies is open source, so off I went with my
desire and used Stickies for Windows as a starting point to make some modifications so it’d do just that.
The fruits of that effort are an application that behaves similarly to sidenote, with my own flavor added.

I am continuing development to add other similar features as sidenote as well, but what’s available works reasonably well enough that others can start playing with it.
You can get binaries/source code/bug tracking etc. from google code, here. It’s a C#/.NET application.

Enjoy! (warning, it needs polish)

Popularity: 100% [?]

May 4th, 2008

Book: Facts and Fallacies of Software Engineering

I recently finished reading this book. See the books page for more details.
In short, it has some interesting information and is certainly worth a read.

Popularity: 85% [?]