How hard would it be to...

Sometimes there's a big difference between the perceived difficulty of implementing a new feature from a developer's perspective and a customer or client perspective.  These are the types of features that sound simple, but in reality are often not worth the extra development time.  Here are a few examples I've run into:

Adding an 'other' option

If you're building a web application, there's a good chance a user will at some point need to choose between several choices in a form.  You might implement this with a select box:

Or maybe using radio buttons:

This implementation is simple, and it works.  The customer is happy, until they think of a potential problem:

Some people might want to choose something that's not on the list.  How hard would it be to add an 'other' option, and let people type in a custom choice?

From the customer's perspective this sounds extremely reasonable.  Why is it harder than expected?  Two primary reasons.

The UI needs to be updated in a non-trivial way

Adding another color choice to one the lists above is (literally) seconds of work for a developer - just copy/paste one line of code.  Adding an 'other' option with a text box is not so easy.  Now the developer needs to add a new input field of a different type and figure out how to style it properly.  If an 'other' option is added to the select box, maybe the text box should only appear after the 'other' option is selected - introducing Javascript to what was previously just an HTML form.

The way the server handles the request and stores the data may need to be changed

What previously might have been a single parameter in the request may now be split into two parameters (the choice from the list, and the text optionally entered in the 'other' box).  If the choice was represented as an enum (and stored as an integer in the database) this data type may need to change, or a new column might need to be added to the responses table.

What's the best approach here?

One choice is for the developer to push back: see how important having an 'other' option really is compared to other priorities, explain that it may be (somewhat) difficult to implement, and see if there's an alternative solution to get you most of the way there (like adding more choices).

Dynamic Form Updates By Administrators

Here's another scenario: a client has a list of fields they'd like to include as part of a user's profile.  This might include name, picture, email address, hometown, and a bio.  The developer builds a basic form where users can enter those fields.  After some additional work, the client realizes profiles also need to have phone numbers, so the developer adds this to the form.  Later, the client decides hometown shouldn't be on the profile, but birthdays should, and the developer makes these updates.  At some point after more back and forth, the client has another request and realizes this process is inefficient:

Let's add a place for users to enter a nickname on their profile.  Also, I might want to keep adding or removing some things to profiles in the future.  How hard would it be to setup a place where I can easily make those changes myself?

This kind of back and forth is to be expected as a platform evolves: consider how much Facebook has evolved, not just with new features, but in the information entered and collected as part of your profile.

In this case, the client, potentially familiar with site builder tools (say Wix, Squarespace, etc.) believes this request is reasonable and will save back and forth time in the future.  Why is it harder than expected?

Forms are not like static pages

Static pages (like homepages, blog posts, other landing pages, etc.) can often be served using a CMS or blogging platform (Wordpress, Ghost, Siteleaf, etc.). The client can interact directly with these platforms to make updates, avoiding the need for back and forth with a developer.

Unfortunately, adding a field to a form is not so simple.  If the client is given access to the HTML code for the page, copying and pasting an <input> tag is going to be error prone without knowledge of how to update the necessary attributes.  In addition, in most cases it will not be possible to update a form without some understanding of how the data is being stored.

In the case of a user profile, does a new field need to added to a User model on the back end?  Does a database migration need to be written to add a column to the User table?  Fundamentally, submitting a form, unlike viewing a static page, is going to change state on the server, and so some understanding of how that occurs is going to be necessary to make updates.

Building a form is nothing like building a form-builder

So why not setup a way for the client to edit the form that hides the hard parts?  In short, this is going to take a significant amount of work and is probably not worthwhile.  At a minimum an entire new interface needs to be built to allow an administrator to add or remove fields from the form, and the underlying data representation needs to change to accommodate these form fields being added or removed dynamically.

Perhaps the best evidence that this is not easy is the existence of specialized form builder tools like Wufoo or Typeform.  Entire businesses have been built to give people the ability to easily create and update forms without a developer.

What's the best approach here?

• Ideally, this problem can be avoided by setting expectations clearly up front.  When working on a new project, I like to make it explicit that there won't be any easy way for administrators to modify forms the way they might be used to with other pages: it will have to be done by a developer.
• In some cases, third party solutions might be useful.  Depending on the customer's goals, even something as simple as a Google Form might work.

Keeping Web Scraping Up To Date

Lots of startups will use web scraping at some point - maybe to gain information about competitors, to populate an initial database of potential customers, etc.  Most basic, small-scale web scraping tasks (e.g. scraping a list of businesses from an online directory) can be completed in an hour or two with a well crafted Python script.

Of course, once a client sees how easy it was to scrape the information the first time, there's a natural follow up question:

Now that we scraped that data from X website, how hard would it be to set something up to check if that list ever changes, and keep the list on our site up to date?

To a client that's seen immediate results from web scraping up front, this sounds reasonable and potentially simple.  Why is it hard?  There are two key issues:

Where will it run?

A one time scraping task is typically a script written on a developer's computer and run by the developer.  If scraping is going to be re-run automatically, say daily, what computer is it going to run on?  Having the developer manually run the script every day probably isn't practical.  If you have access to a server you might be able to set up a cron job, but if you're using a PaaS like Heroku, that's going to take a little more work.

How will errors be handled?

Sometimes websites are down, and sometimes they change how data is formatted.  For a one time scraping task this isn't a problem - the developer can look over the results and make sure that they look OK.  If scraping is automated, and the site happens to be temporarily down when scraping runs, will you end up deleting records in your own database when syncing the results? How will you get alerted that there was a problem?  Suddenly your simple Python script needs to track errors and send out email alerts.

What's the best approach here?

• One again, try to set expectations clearly when scraping the first time: explain that scraping a website once does not mean it will be easy to automatically scrape the same site daily and keep the results in sync.
• How often does the data really need to be re-scraped?  If you just want an update every few weeks, then having a developer occasionally re-run the script might be practical and cheaper than building automation.
• If automated web scraping is an essential part of the platform you're building (say, a price comparison platform), think about error handling and other infrastructure from the beginning.  I've had great experiences with scrapy.

Conclusion

When a developer is working together on a project with someone that doesn't come from a software background, part of the job is to know when to push back and see if the underlying goal can be achieved in a simpler way.  During recent consulting work for several startups, I've found this to be essential.  Really understanding a client's goals can tell you what code can be left unwritten, what parts of the platform can be outsourced to third party tools, and what processes can be run manually instead of implementing unnecessary automation.