small problems with alphabetical sorting and with autocompletion

[Anson]

problem with alphabetical sorting

When the list of movies, actors or some grouped listing are sorted, it is done strictly according to the charset (ASCII?). This results in listing all uppercase titles first, followed by lowercase titles. While people use the complete titles without moving "The" to the end of it, and while using english titles (usually all words are capitalized), this doesn't matter much, but when there are lots of German (or french, italian, etc) titles, and/or when moving articles to the end, many titles suddenly appear at the end of the movielist instead of being alphabetically sorted together with the capitalized words. Similar also applies to special characters like "-", "'", "(", etc, and also to special local characters like äöüÄÖÜß, èéêùúû, etc which partially appear before uppercase, between upper and lowercase, and after lowercase.

It would be nice to be able (either by default or selectable by an option) to sort all special chars together at one place (either before or after all other chars), and most of all to sort chars according to the local language or in some other merged way, at least ignoring upper and lowercase, or even eg AaÀàÁáÄä together, followed by Bb, etc


problem with autocompletion

In my database, I have added several custom items, including "select list". When I want to select a value in such a list by typing the first few chars, two effects occur:

The chars i type are taken literally including uppercase and lowercase, and autocompletion only takes the rest of the string from the selection list. Since the typed chars are taken as typed but it looks as if the field was automatically filled with an existing value, I already ended up several times in unintentionally creating a new value which I couldn't easily see (because of the sorting problem, see above).
Example: A value "Disney" already exists in the selection list, but i simply type "d". Autocompletion makes "disney" from it, and when i accept, a new value is created. Sorting or grouping by this field will later cause this new value to show up at the end behind all uppercase values (A ... Disney ... Z ... disney) and the record seems to be missing in the database when i look at the "Disney" group.
It would be nice when case would also be adjusted automatically, although i recognize the problem of "how to add a new value starting with different case if a value starting with those letters already exists"
I have no perfect solution for it, but wanted to explain the small problem i have with the current situation. Maybe in the meantime, autocompletion could match the exact case only, which would require a little more carefulness (also using the shiftkey instead of "just typing"), but avoid unintentional additional values.

Another problem with autocompletion is a bit more strange:
In a custom select list, i have (amongst others) the values "Horror - King", "Stephen King", and "King Kong". And here is what the autocompletion does:
- on typing "step", I get "stephen King"
- on typing "horr", I get "horror - King"
   (both as expected, with lowercase problem: see above)
- on typing "stephen ", I get "stephen "
- on typing "horror ", I get "horror "
   (both: why is the rest no longer completed when i type the space?)
- on typing "stephen ki", I get "stephen king Kong"
   (rest is completed again, but: what kind of completion is THAT?)
   (at least i had a good laugh on Stephen's new nickname :-)
- on typing "horror - ki", I get "horror - ki"
   (rest still is no longer completed after i type space and "-",
   and it is also not even completed as above like "horror - king Kong")

Is the autocompletion done by PVD by doing some patternmatching, and the regex and stringoperations cause also unintentional matches like the "Stephen King Kong"? When i programmed something with a listbox in one of my programs, i set flags to ignore case and to alphabetically sort the list. When I type something in the editfield of that dropdownlist now, it doesn't show autocompletion, but opening the pulldown or using wheel or cursorkeys shows the first value which matches these typed chars, making it easy to use mousewheel or cursor-up/down to select an existing value (including the correct case of the characters), and the preselection of a value speeds up scrolling to the intended value, most importantly in very long lists.
Of course, this is only my personal approach in one of my programs, but for my style of working, it worked out: type a few chars (without watching for correct case), use the wheel or cursor (even a single "down" will do), and get to a correctly spelled existing value fast. For these abilities and less other problems, i would gladly sacrifice a shown autocompletion :-)


ps: another small general problem with the select lists is that after editing a record, typing a new value for a select list, and applying the changes, this value won't be considered for autocompletion on editing another record later, unless i have first clicked the arrow to pulldown the selection list at least once: The list of values seems to not be updated automatically, and i have to force such an update by showing the list manually once.

[rick.ca]

Sorting of data is done by Firebird, not the program.

The primary purpose of the program is for managing downloaded data. For the relatively rare cases where data entry or the changing of downloaded information is required, I think the list behaviour is fine as is.

[Anson]

just like with my other post, you picked those parts which are less important, and didn't say anything about the other parts, eg what you think of autocompletion not working when it hits on "word - anything" or that it generates autocompletions like "Stephen King Kong" ... I don't want to "attack" someone or something, and that i use the program should be proof enough that i like it in general, but only commenting on some points which you find "fine as is" and not saying anything about the others, might give the impression as if nothing I wrote would have some real base for reporting.
even when i can live for now with those glitches (or eg autocompletion generating new values with different capitalization instead of selecting only existing values), shouldn't I report them to be addressed in some future versions ?

Sorting of data is done by Firebird, not the program.

I don't know Firebird, but are there no options to tell Firebird some details like numerical or alphabetical sorting, sorting by local charsets instead of ASCII/Unicode, maybe even specify a custom sorting procedure, etc ?

The primary purpose of the program is for managing downloaded data.

including to retrieve data and look up movies, use groupings, sorting on some field, etc !?!

For the relatively rare cases where data entry or the changing of downloaded information is required, I think the list behaviour is fine as is.

as i said in the subject line: "small problems"

i think that data entry or editing is not as rare as you think. I myself will have to edit each movie record, eg to modify the (localized) title, to add info about my sorting (like movie groups, etc) and my storage folders, and to add info on the type of media/DVD/file/etc, for a total of 15000+ fields (including 9000+ select lists) on the first 3000 movies. And that has nothing to do with list behavior, but with the autocompletion.

the normal use (looking up movies, displaying groups, etc) is related to "list behavior" and as soon as I have localized all titles, my guess would be that half of them are lowercase and half are uppercase. together with possible errors from wrong capitalization on select fields (and i wanted to use select fields instead of short text fields just to avoid typing errors), I don't consider it "fine as is", but only "can live with it"

[rick.ca]

...shouldn't I report them to be addressed in some future versions ?

Sure. But if you do expect these things to be addressed, report them in a manner that respects the fact there are very limited resources for doing so. So, list autocompletion, for example. While the facts you've reported are probably correct, it's a very minor issue. If it's going to be addressed, it will be because your concise post is readily understood and recognized for what it is, added to some todo list, and then considered along with other issues and suggestions when that part of the program is next reviewed for possible improvements. If it takes nostra longer to read, assess and debate a suggestion than it would to simply implement it, it's not likely the suggestion is going anywhere.

Having analyzed it so thoroughly, you know full well the list autocompletion matter barely qualifies as a bug. It's probably not affecting anyone else, or they would have said so. Now that you understand the behaviour, you can easily adapt to it. So, no immediate action is required, and you're not in need of assistance. This is a straightforward suggestion for the improvement of a program function that should be posted as such in Feature Suggestions. You may do so any way you like, but I'm sure you'll find the more concise the better. If nostra has any question about the suggestion, you can elaborate then. If your suggestion is readily understood, you'll also find other users will add their support. You'll know you've communicated a good idea successfully when they simply post "+1."

The sorting issue is another example of something that has been reported and discussed in the past. I don't expect everyone to find such previous discussions, but you obviously have no aversion to research. Directing a little bit of that effort towards the use of the forum's very effective search function would be worthwhile. An understanding of where a particular issue stands will put you in a better position to contribute something new and useful to the discussion. Even if you don't find past discussions about an issue, try to respect the fact some things are so obvious they must have been discussed before. If that's the case, it would be more respectful to lead with a question (e.g., "I just noticed [this behaviour]. Is there a reason why it works that way?") before launching into your own problem analysis and recommendations.

[patch]

Anson, your analysis of the problem is clear and thorough. I agree it is a deficiency in PVD which significantly impacts on the ease of use of its user interface. Improving the sort order presented to the user would be a useful enhancement.

It is true it has been superficially discussed in the past.

Sorting of data is done by Firebird, not the program.

Is rick.ca summary. Which I believe is misleading.
Firebird could be programmed to present the data in what ever order was desired. See http://www.firebirdfaq.org/faq244/
The valid point is it is not the default firebird sort order.

The primary purpose of the program is for managing downloaded data. For the relatively rare cases where data entry or the changing of downloaded information is required, I think the list behaviour is fine as is.

rick.ca This comment comes across as: "I don't want to use PVD that way so nostra should not waste his time doing this and instead do things which are important to me"
In the interests of encouraging broad discussion as opposed to argument, please consider first exploring other users perspective prior to writing it off. You never know, a new forum member may go on to make a valuable contribution to PVD.

[rick.ca]

You never know, a new forum member may go on to make a valuable contribution to PVD.

You're missing my point entirely. Anson has demonstrated his ability to thoroughly test and analyze the behaviour of the program. If I didn't believe he could make a valuable contribution, I wouldn't waste my time pointing out how he might present his findings and suggestions more effectively. It not effective to ignore previous discussions about an issue (e.g., sorting of data is done by the program). It's not effective to make a big deal of a minor issue so the developer has to spend more time sorting out the controversy than he would just making a simple change or fix. It's not effective to criticize a program function outside it's intended purpose (i.e., if you don't agree on the purpose, make that the issue).

Speaking of being effective... You imply a resolution to the sorting issue is not hampered by the fact sorting is done by Firebird rather than the program. I'm sure most users would welcome a fix if one is available. But you also indicate you're aware of previous discussions about this, so you know my "misleading" response is based on nostra's last word on this. So what is the purpose of your comment? It hardly seems a productive approach to guiding nostra towards the resolution of a long standing issue that would clearly be a valuable contribution to the program.

[patch]

The order data is presented from a program can always be controlled be the programmer in any general purpose programming language.
What varies it the effort required to achieve it and the benefit to the application functionality.

So forum discussions can usefully address the benefit to a variety of users of a feature suggestion. Anson has done a better job than past posts to elucidate the issues, & I thought deserved support.

As for the difficulty in implementing a solution, only nostra can judge that but firebird does offer some native support:
Case insensitive sorting has been added to firebird 2.1 http://tracker.firebirdsql.org/browse/CORE-972
There is also the ability to support character set ordering other than the default ascii order http://www.destructor.de/firebird/charsets.htm

Given nostra past comments I suspect he may consider it at the next major program revision, depending on other priorities, feature popularity and implementation difficulty.

[rick.ca]

As for the difficulty in implementing a solution, only nostra can judge that but firebird does offer some native support...

Thank you. This moves the subject forward. I have no idea where to, but it seems to be worthy of nostra's consideration. If not, it's at least a legitimate question.

I note from the reference provided the "solution" seems to be language-specific. That suggests to me it's a lot more complicated than just modifying the code to take advantage of a new feature in Firebird that does the sorting "properly."

[Anson]

I note from the reference provided the "solution" seems to be language-specific. That suggests to me it's a lot more complicated than just modifying the code to take advantage of a new feature in Firebird that does the sorting "properly."

Each language and each country has different charsets and sort orders, and there even exist several different sort orders for german lists, eg in phone books vs. other official lists etc (eg whether spaces, hyphens and similar are counted, whether Ü becomes Ue for sorting or is sorted between U and S, etc). As long as sorting is done for one language only, it still is relatively easy, but when international movie titles are concerned which are given in many languages in a single list, all (contradicting) sort orders might apply at once. Those are the problems ...

But as long as it is used mostly eg on localized titles, the user's local language scheme might be a choice for basing the sorting on it, and even a default mapping of lowercase and uppercase might improve finding titles. I don't know about Firebird, but other systems allow a choice at least between case-significant and ignore case, or sorting according to no special charset (would sort according to the program's charset), according to an internal simple lower/uppercase merge, or according to a selection of the user (either the user's installation language or a freely selectable language).

For my purpose of not finding accidental "Disney" and "disney" at almost opposite ends of the list, any such method would do :-)


over all this talk about sorting, please don't forget that my main point should have been the problems with autocompletion:

According to what i was taught, some purposes of select lists (eg instead of using short text) should be to enable users to only use a specific set of values and to handle (select) them faster instead of typing strings slowly (this is the user's side; I won't speak of more reasons like internal management, enumeration lists, etc here). The autocompletion itself is only one (but the one nostra has chosen) of several methods for selecting values from that given set of values, and thus should not cause the set of values to be modified. And the ability to enter new values directly in that field's box (instead of having to edit the set of values in preferences or similar) is only a convenience (although an important one) to users. The fact that autocompletion in PVD is used to select existing values as well as create new ones makes it more difficult and gives lots of possibilities for problems, but nonetheless those basic principles should still apply.

Secretly modifying this set of values in any way (besides non-obviously creating new values with different case which I find highly disputable), not finding existing values and finding not-existing values defies all that and thus is a real bug in treating select lists.

While I still have mixed case in my field values and with the additional problems, i even would prefer to have no autocompletion at all (selecting values from the pulldown list instead). For me, all this is no "program stopper", but a part of the program i can't use as intended. A "feature suggestion" would only be a discussion about how to handle select boxes differently, but my first concern was that select boxes are handled correctly (either by fixing the current method or using any other method).

PS: I had written a lengthy answer to the previous 4 or 5 posts, but since the topic seems to advance a bit, I won't blow it up now here. The above was the most important, and I'll send the more lengthy reply as copy to rick and patch only