Rethinking form validation
We all know how important form validation is. Users must provide required information, users must provide information in specific format, user must… Right. But imagine the scenario in which user saves a form even if validation fails. That is, when inappropriate formats are provided or required fields are left blank. Sounds ridiculous? To kill validation? Well, no – we just might rethink the validation process.
My team had a debate about this topic when we first read Alan Cooper’s suggestions on data saving. We decided to try it in one of the projects. Developers hated it. Designers endorsed it. Users loved it. Let’s see why.
How should this work?
Here’s the scenario: user, who works in one of the bank branches, fills in the same forms every day. She deals with different forms, from simple to complex, such as leasing requests. If a customer doesn’t provide all the information necessary to approve the leasing request, which happens often, she won’t be able to submit the form and store it in central database. Sounds correct? Let’s try again.
User, who works in one of the bank branches, fills in the same forms every day. She deals with different forms, from simple to complex, such as leasing requests. If a customer doesn’t provide all the information necessary to approve the leasing request, which happens often, she can enter whatever information she has at the moment and store it in the central database. If some of the required fields are missing she is notified with inline validation that she needs to provide it later in order to process the request. If she enters some information in the wrong format she is also notified by inline validation that it should be corrected in order to process the request. But she saves the request. Incomplete, but she can save it.
User becomes more confident because she feels that she can’t make a mistake. No errors, no popups, no interruptions. She can easily save an incomplete leasing request and complete it later. When she opens the request again inline validation will show which “required” information is missing and what data is in the wrong format. Of course, the leasing request can’t be processed without information that is absolutely needed for calculations, so it can be processed once the form is completed.
Is required really required?
I deliberately placed the term required in quotation marks here, because required information isn’t required in the way we perceive it. Information is required for successful leasing request approval but not for storing the same request in the database. Same applies to data format. So, think again why some information is required – to defend database integrity or to support users’ goals.
Don’t take this literally
This can’t be applied to every single form. Of course, this makes sense mostly in so called enterprise web applications. For instance, it wouldn’t make much sense to submit a comment on my blog without providing the comment and name. So I’m not suggesting that you should make the worst example of web form validation ever. You get the point.
One thing is certain – this would be a headache for developers. Not only because they love to defend the database in any possible way, but also because such approach would require much more work on their side. But more important is that there would be fewer headaches for users.
* * *
Remember RSS? You can subscribe to my blog here.
44 Comments
Actually, I think this -would- cause issues for users. I can already hear the complaints.
"I submitted the form, but it never processed my request!"
… Yeah, because they ignored the yellow box telling them that it was doing so. All they’ll read was that it was saved.
And yeah, we like to protect the database. If it’s designed to hold partial data, it’s not a problem. It very rarely is designed that way, though.
I agree that if you’re working, you should be allowed do as you prefer, entering all the information at once, or in chunks. In e-commerce, many fields are for marketing purposes only, so they should be optional.
I don’t care about required fields, what bothers me is too long forms. The name in a comment form can be optional too, anonymous is better that a lie (like "asdf").
Great post. A web app that the company I work for creates is a business application that can sometimes require 100s of fields to be filled in a form, but almost nothing is required before saving, even though many, many fields have various validation requirements — we allow saving with errors (only critical errors, which are rare, cannot be saved). This lets the user fill in what they can, and if needed, research the answers to other fields and come back later and sort it out.
This has been part of the product since its very beginning and now that it is baked into the code from the onset, it is accepted by developers quite well. In fact, it was a business analyst who first joined and saw this that struggled to accept it :)
Interesting post. I’ve seen this concept a couple of times already, in online banking for example. You can fill in application forms and save them to return at a later stage and complete it. The possibility to save a temporary state makes great sense for large forms that go over several pages. But I agree, for feedback forms it doesn’t make sense.
I completely agree with this post. Of course you should make sure that saving partial data has value/meaning in your application’s domain. I’ve found that it often makes sense to do the validation on a state change, rather than on saving. For example, an application can’t be changed from the "Pending" to "Submitted" status without fully passing validation, but there’s no limit on saving partially filled forms.
To rebut @WC’s point, if users are confused about why an application wasn’t processed, you need to provide better feedback in the UI. Consider how anytime you do something in Gmail a notification bar shows up near the top of the page. (Like after you send a message, you get returned to the Inbox and see a "your message has been sent" message.) We’ve implemented the same concept in our applications using JBar (jquery) for notifications. Successful actions show a green message bar, unsuccessful actions show an orange message bar. Also, having a landing page that shows users pending applications or warns them about applications that have been pending for too long would help.
Here is the customized JBar implementation that we use:
http://geekswithblogs.net/ryanohs/archive/2010/05/21/customizing-jbar-for-notifications.aspx
I think this raises a good point about validation but as you rightly point out, it’s probably more use in the "enterprise" level where forms are much more detailled and often go uncompleted at the first sitting.
I don’t see any reason why you shouldn’t be able to save an incomplete entry – been trying to think of one I saw a while back which did this well but I’m struggling for the url.
It’s a good idea to save partial data. No one wants to come back and start all over. Of course it’s more work for the developer.
This brings up a few questions:
1- Why do you have more than one Save button? Why not one Save button for the entire form and make it simpler?
2- Do they have to log in when they come back or are cookies used or ..?
3- Do you send emails reminders when they haven’t comeback after some period of time? I guess maybe you require email field to be filled before they can save the form so that you can send reminders?
4- For the chance that some users might miss the yellow box and the fact that the form was saved and not submitted for processing, maybe a popup, in your face, type of message to make sure they get this piece of fact.
Thanks for this interesting article!
Submitting an incomplete form in to a database is not always possible. For example, what if another system access the same db and acts on records there with the assumption that they are complete. Now you have a problem.
In that case, a more elegant solution is to have a Save button and a Submit button. When everything is filled out properly, the submit button is accessible. Otherwise, only the save button is used. The saved information can be stored in a temporary db, file, etc until the rest of the information is provided later.
There are a lot of restrictions for either scenario (yours or mine). But you’re right – you should be able to save data for later when the situation permits.
A proper feedback should prevent this to happen. As Ryan mentioned in his comment below, different colors might be one of the ways to handle it. Also, users work with line of business applications every day and as majority of them are intermediate, they rarely miss such feedback.
Also, the software can inform users about the state of data in many ways. For instance, when presented with the list of leasing requests, software can easily indicate which requests are incomplete and which are not.
Exactly. Having [i]pending [/i]and [i]submitted [/i]states is one of the ways to handle this.
If you remember the url, please share, would like to see it!
Abdu, the image represent different states of the same form. So there is only one [i]save[/i] button.
Regarding your second and third question, it depends of the domain, requirements, etc.
For the last question, I would avoid popups (check out my previous article http://www.jankoatwarpspeed.com/post/2010/06/17/modal-overlays.aspx ). There are plenty of way in which software can inform user about the state of the data: proper feedback upon saving, inline validation in edit mode, list of stored information with clear indication about their statuses, etc.
If another system is using the same data it can easily see which data is incomplete and treat it accordingly. I wouldn’t add more actions besides save – it would add unnecessary complexity and might even break users’ mental models: they just want to save the data, incomplete or not.
Very interesting concept and post. It is often difficult for my employees to complete certain form I use in my business, leading often to me having to doing their job because there’s simply no leeway for them. I would be very interested in updating the way I do my data entry with an application which is modeled after your concept. How feasible is such an application? I am not sure the "save" vs "submit" option – wouldn’t the saved version only be available on a single workstation rather than to every user of the database?
I understand the point of your article, so I won’t digress much, but presuming another system can detect incomplete data is a massive fallacy. I’m sure you know this can only be presumed if you’ve thoroughly tested the system or have the source code in hand. I can think of many scenarios where this would not be the case – especially if the original system was not designed for this type of partial submission. Unless everything accessing the same data has been designed with this in mind, it can be a lethal trap.
Either way, I like the article and as always you’ve got excellent points/thoughts.
Nope, the "save" feature would still save the record to the database. The only consequence is that you can’t have required fields in the DB anymore (except ID, or things that would always get assigned). You’d want to validate that all the required fields are filled in the code at "submission", not when the user clicks save. If the database is your integration layer to other applications, you’d need to have a separate table to store "in progress" forms, or have a way to mark a record as complete.
On the client side, you could have some javascript that would show either a "Save work in progress" button or a "Submit completed application" based on whether all the fields are filled in. However it might also be nice to display the save option all the time. It really depends on your scenario/use case.
It depends on the scenario. If the application is a part of the integrated system with shared resources (office suite for example) then it shouldn’t be the problem. In the other hand if original system is accessed via API, it can provide a flag that will determine the state of data. In any case, at least some changes in other systems would be necessary.
It’s always debate, whether to have statuses in one table or to have two tables with consistent data (one with incomplete requests and one with valid data). Either case has pros and cons, it’s just on the team to decide which one will be used.
But User Exp should be the same, to be able to draft save the request, if that is the scenario we want to achieve.
hey janko, i think these scenarios are completely different entities.
as an admin, this is exactly how the ux should be.
as a user filling out an eCommerce form, as noted, this would be awkward.
I think the easiest way to have the best ux for these types of forms would be to only have the user fill in what is absolutely needed to process the form.
An interesting idea – more suited for backend systems and the like rather than front-line use by the general public IMO.
UI feedback is discussed above, but in my experience if you are working with the general public, there will always be a certain percentage who fail to read ANY instructions, notices etc, and then proceed to get shirty because things haven’t happened as they expected. They tend to be people with "@aol.com" email addresses too, funnily enough… :)
However, in an application where the users have a little bit of knowledge, this would be great, particularly in a collaborative environment where there might be two or three people needed to fill out all the fields.
Exactly, it doesn’t matter for user how it’s implemented in the back end.
Absolutely, I think its place is in line-of-business applications.
Concerning feedback, users of line-of-business applications are mostly intermediates who have domain knowledge at least to some extent, so I don’t see a feedback as an issue (unless it’s poorly designed).
Designing collaborative system in such way is a good idea :)
Yeah, that was my line of thinking – business users would have more knowledge in the first place, and you can easily train up any stragglers, so it makes the UI less of an issue.
Another interesting aspect of this discussion is a paper form processing. Sometimes it is by law that people must fill in paper forms first which needs to be signed. Later these forms must be entered into web application for further processing. 8 years ago I would replicate paper to web form 1:1. Now I think a smarter way will be to enter key information and allow users to scan paper form and attach it to a form. Why to waste time retyping lengthy description fields if they are not required for application business logic?
Regarding your post. It depends on logic of your application and … passion of a developer. Did you see many people developing business applications (not Silicon Valley start-ups) who are passionate and excited?
1. Long form and user can be interrupted – need to save half way through
Even a few key fields – normally top ones will be enough to save just to be able to INSERT into database.
2. At a time of entry not all data could be available but all data are needed to start processing
If it is possible you just require to enter key data – reasonable information which should be known at a time of a record creation, you allow to save it in a pending status and if some required data still are not entered, you remind user about pending records by threshold 1 (5 days), you notify users’ manager if pending more than threshold 2 (10 days) and you keep some fields as not required at all as otherwise people start typing -‘SDDASD’ , ‘n/a’, ‘NOT KNOWN’ etc.
General rule which is so hard to convince business customers, especially in a big corporations – do not create fields if you are not going to analyze it or it is required by law.
Hey Igor, you made some good points. I’m quite familiar with ‘ASDF’ and similar entries :) It just proves that we should rethink this model.
Actually it would be quite funny to compile a list of the best nonsenses people typed, but contracts make it impossible.
Excellent idea! Currently it is a setback for website designers to make application forms full of required boxes. It seems the more boxes required the more website viewers unwilling to fill the forms– the database integrity in exchange of users’ goals? A bad deal for sure. To simplify or at least to enable the saving of incomplete forms is beneficial to users and owners both actually.
Thanks for sharing your view. And the worst example of web form is quite illustrative.
Very helpful post. Thanks a bunch, and please keep em coming! :)
Good article!
It’s like a middle tier before processing, though there are a couple questions to be answered: 1) how many form fields are needed to be filled in before a partially filled out form can be saved? 2) How long do you save these incomplete forms (are you to hound them weekly to finish them)?
As for a [[b]submit[/b]] and [[b]save[/b]] buttons it can be a single button: required fields should be highlighted, marked required, etc. When all the required fields are filled in and validated, the button changes from [[b]save[/b]] to [[b]submit[/b]]. When the user clicks save, the user should be notified that the form is incomplete. If the button now shown as a submit [i](especially on a lengthy form)[/i] the user should be asked to validate their submittal before processing and if the user is unsure of their entries, they should then be given the option to save and submit for processing later.
I had never really considered the possible benefits from the users perspective with regards to creating a form in this way. It is certainly an interesting idea and I can see how for some sites this would work really well. Thanks for sharing your thoughts.
Good questions. I think the minimum fields should be truly mandatory – for instance, only personal ID. The answer to the second question depends on many factors, from business requirements to various constraints. But they should be kept in the database for as long as possible.
I agree that submit and save should be one button – no need to add complexity.
Keep on the good work! Very helpful post. Thanks a bunch, and please keep em coming!
The last comment:
[i]"One thing is certain – this would be a headache for developers. Not only because they love to defend the database in any possible way, but also because such approach would require much more work on their side. But more important is that there would be fewer headaches for users."[/i]
Many times there is more than one tier, the dba’s, the developer, and then a designer. The designer can see the the "ideal" environment, the dba’s see’s a leveraged data pump, the developer is caught in the middle with a balancing act. Users hate entering creditials (required info) – web designer’s focus, dba’s dislike storing incomplete info (no tuning), developers (middle tier) try to put in place the mechanism to satisfy both ends.
Yes dba’s have a point, poor performance and the designer’s point, of long boring, redundant forms the user sees, both will kill hits. This may be harsh but the dba and the designer are many times pigeoned holed on their focus. A collaboration between dba, developer and designer is needed, focusing on trimming the fat (dba), eliminating the nuisance (designer), and making things work (developer).
The ultimate outcome is not satisfied by pursuing a particular focus but a blend of all involved in delivery of the final product.
I agree that in some cases, this will be a solution to lots of unnessecary annoyance, except for information that really is ‘required’ :-)
This makes the most sense to me. If the known data could be saved in one file and only submitted when everything was completed, isn’t this the desired end result? No DB corruption, and a ‘reminder’ email can be triggered off every week to remind the people requesting the service that they still had to complete their application.
There’s nothing more frustrating than submitting a form and being given error messages once the page reloads. JavaScript solutions help to avoid this frustration, and in many cases it’s enough to encourage the user to correct the errors.
In an enterprise environment where data is being entered into a form, a save in progress option would certainly be useful, but I agree with some comments earlier, that not everyone will understand that Save does not mean Submit. Coloured error/incomplete status messages would need to be intuitive enough for new users to understand the status of the form (I.e. whether it was simply Saved for later or Submitted for Approval).
Saving incomplete forms should not be nothing new. In Finland you can do the tax form online and they make it perfectly clear how to save the form at any time and how to submit it later. There’s definitely a place for saving incomplete forms.
However, you probably don’t want to put a save-a-draft everywhere, like really short and simple forms in public systems like online store registration form or anything else cookie-based. You just can’t save a person’s data without being sure who’s the person. In many countries, there are also strict laws in place that put some constraints on what you’re allowed to save and how you keep your data safe. For example, you need to have the person’s consent on your terms of use etc. So, authentication needs to be done before you can go saving incomplete forms.
An example of bad forms design from a Finnish bank years ago: Clerks needed to create dummy accounts (for Donald Duck and Mickey Mouse etc.) in order to show options for potential customers. The system just didn’t allow showing options to customers before they had user accounts in the system. There was incomplete form saving, but it was a wrong time.
Hey Jarmo, thanks for sharing your thoughts and examples!
Very nice redesign. I learned about it in your feeds, and came to see for myself.
In this age of visual clutter and bling, its nice to see something simple and effective.
Mistake: the correct spelling is intergalactic, not intergallactic.
Thank you Thomas!
typo in the graphic: furhter
In order to process it further…
oh, and the comment form wouldn’t let me submit without entering an email ;) A valid email.
Okay, I just digested this and you nailed a system I’m about to implement where I’d like my users to get in and get started but I need them to have logins and valid addresses. Thing is, only the login is necessary to interact with the system. The address is only necessary when they execute a transaction – that comes later. In considering your post, it’s a great thought. I really would prefer they not get stopped at the door and to clean up or press them for an address later when it’s truly needed.
Thank you for this!