Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.


Messages - Anson

Pages: [1] 2 3
1
Talk / Re: Favourite Website / Tools ? (Movie / TV related)
« on: February 08, 2010, 10:46:51 am »
... more ... highlight ... right click in window for options ... Select remove or hit the Del key

I asked this question because I already had tried that, but nothing happens when i rightclick (in the list of files) or hit del. And when i click anyplace else in the window i get the context menu which has options like "always on top" and "exit". And i did NOT push the button labeled "delete" since i think that will delete the source files and not only their names in the list !?
btw: i have only the free version 2.12 and not the pro version.

When i just started to move files, i got the error that more space was needed. thus i wanted to select a different destination, but clicking on the destination path and selecting a new path was only possible after i had clicked the "ignore" button. maybe Teracopy is a bit selective (and doesn't tell me clear enough) in which states which options can be used ... Thus i went back to trying to modify the list of files like above and also did the variations of having Teracopy active or paused (and even completely done with copying) while rightclicking and always the same result (no reaction). The only way to remove files for me is to use "clean up", but that only removes all files from the list which were handled properly so that "problem files" can be tried again.

ps: to make it clear again: i like the program a lot and even with limited options to modify the list of files to be worked on, i find it a lot better than the builtin copy/move in vista (which has no such options at all, no pause, no queues, nothing of this).

2
Talk / Re: Favourite Website / Tools ? (Movie / TV related)
« on: February 06, 2010, 12:31:19 pm »
Quote
(*) small problem: when I started a few dozen copies at the same time, the last seems to not have seen a running process and thus started automatically, but below 20-40 parallel copies, it always worked perfectly :-)
You can drag and drop onto an existing teracopy process and it will add files to be copied to the end :)

Yes, thanks for the hint (btw: how can i remove files from such a list?).
But that is only usable when you want to copy more files to the same destination, and I got the 20-40 parallel processes from copying movies from one temporary directory to many different genre-subdirectories on a second drive and then moving the same files to a third drive (one copy as data, one as backup).

What you included in the quote above was intended as a small warning (someone might forward it as bug report to the creators of Teracopy) that Teracopy might not see that there is already a running process and thus automatically start a newly created copy/move if there is already a large number of waiting processes.
This problem is not very serious in normal use since it appeared only after i created a few dozen copy processes at the same time, and in most cases also wouldn't hurt if it happened. But users should be aware of it IF they start a large sequence of copies and moves and the order of executing them matters. eg when first a copy and later a move of the same file should be done (eg to have one file on a data disk and a backup copy of it on another disk), or when first some files have to be moved away before other files can be copied to that free space. only in those few cases, the user would have to abort the last (automatically started) process to not mess up the execution order of the process queue.

3
Feature Suggestions / Re: Remember last choice when doing a bulk download.
« on: February 06, 2010, 11:21:18 am »
It is not really a bug as PVD has no way to know if the 2 digits should be interpreted as 19xx or 20xx.

true, although in the special case of someone born in 17, it can't be 2017 (yet!), and/or the given age of 7 should be -7 (minus seven).

Such an additional check for negative ages might temporarily help a bit but probably won't be worth the effort to implement, since sooner or later there will be many more movies and actors with dates that are more than 100 years ago. Maybe it could be useful to have a hint somewhere (in a manual or some installation instructions) which reminds people to be aware of this problem.

btw: just a month ago (on the change from 2009 to 2010), tens of millions of bank cards stopped working in germany since their builtin chips had a software error which didn't handle the transition from 9 to 0 ... although those cards were not free like PVD and although (after all that talk about the millennium bug 10 years ago), that transition problem could have been avoided easily.

4
Feature Suggestions / Re: Loading database when program is launched
« on: February 04, 2010, 10:54:58 am »
As I've explained here, I don't believe there's any reason to be concerned about the size of a database due to the images stored in it. I think you've created a huge inconvenience for yourself by splitting your database—at least the separation of movies and series. The program is carefully designed to handle these two video types together in one database. It's obviously much easier to maintain one database than two.

I agree in general that people should use as few databases as possible to avoid the additional work of maintaining several versions of templates, user defined fields, etc, and to be able to have the advantages of a database with links back and forth between movies, actors, other movies, etc.

BUT (at least in the current version and how i use PVD), movies and series seem to work not too well together:
Links to episodes of series (eg from "movie connections") do not point to episodes below a main entry for the series, but to "main movie records", and after updating such series episodes with their title and a localized title, even the special formatting which IMDB uses to point to an episode of a series will be lost, making it very hard or impossible to recognize such entries as an episode (and to which series it belongs) instead of appearing like a movie. And updating a series to include new seasons or episodes from an internet source (without influence on already downloaded or changed episodes), or importing my own list of titles and localized titles for a series (from CSV or similar) is no trivial task either.

Thus in my database, I didn't modify anything in a series yet, and might create a second database to do such modifications not in the main database but only in a separate subset database which will be easier/faster to recreate than the entire database if/when problems get out of hand.

Quote
... (images in the database) ... Because of this, my database is becoming pretty huge (around 500 megabytes at last count).
There's no reason to be concerned about the size of your database due to the images that are being added to it. The images will take up as much room in the file system as they do inside the database. And I don't know for sure, but I suspect Firebird can manage them more efficiently than the file system. They're certainly a lot "safer" inside the database.

true: the total size of the database itself plus the size of all pictures would be the same in both cases, maybe even a little less when everything is put together in one file since the database might handle storage more efficiently or at least differently than a filesystem with given sector sizes and thus rounding up the size of every single file.

but also false: depending on the filesystem, it is a huge difference whether i have a database of a few dozen or even a few hundred MB and separate 4000+ pictures of 1 MB each, or a combined database file of 4+ GB which can't be stored on a FAT32 volume. Even when most computers nowadays have a version of Windows which fully supports NTFS (or some other operating system which can be upgraded with drivers to use NTFS), most movie players (i have an emtec movie cube) don't use windows and (almost) can't be modified in any way. Thus they support at most reading from NTFS disks, but otherwise require FAT32 and/or files smaller than 2GB or 4GB. And thus i now keep newly bought external harddisks formatted with FAT32 (to use them on my comp as well as with other devices) and would like to keep the largest files below 4GB (to store a drive specific database on the drive itself, or to back up a main database on any drive).

I read somewhere that it wouldn't be possible to undo the option of including pictures in the database. Is this still true and/or will be true in future versions?

As long as the database is small enough (less than 2GB or 4GB), I would like to follow the advice to keep pictures in the database, but I need to be able to later have files of no more than 2GB or 4GB in size, eg by extracting pictures to separate files and purging them from the database file. Another reason to keep pictures separate might be to view them directly on a media device (i don't have experience yet on how to do this best).

5
Talk / Re: Favourite Tool Teracopy
« on: February 04, 2010, 07:50:17 am »

... Teracopy ...

Thanks for this hint. It's really useful for me.

I observed only a 10-15% improvement in speed moving large files to another partition on the same drive.

When i copy only a single file with Windows Explorer, I can keep on working at the comp or get a cup of coffee and it's done when i return :-)

Teracopy starts to be really useful when you do several copy or move operations in a row, most of all when moving files which shall be copied before the move, and when copying/moving very many or very large files or from/to many different directories, which would slow down or even can crash the windows explorer, and after starting a copy/move in windows explorer, I would have no control over the process (besides aborting it completely; no options to start later, pause, modify(!) or resume)

Since this is the talk forum, I'll give a few examples how well it served me on using it for the second day :-)


Example 1: i just had 30 movies (around 60 GB) and wanted to move them from one external "working data" USB drive to another "storage data" USB drive, using different destination directories to sort them by genre etc. When I do this with the Vista Explorer, I end up having around 15 copy processes running parallel and sharing a bandwidth of at most 6 to 7 MB/sec (at first, and most of the time dropping to shared 3-4 MB/sec or even lower over time, sometimes even getting timeouts, terrible lag on the entire comp, etc, and doing a single copy operation for 200 files with 400 GB is almost impossible with Windows). Now I did the same (mark files and drag to destination), and this started Teracopy 15 times, with the first of them running and the others being in a state "copy waiting". If needed, i can manually "start now" any of those waiting processes and/or "pause" any of the running processes with a single button, and the first of them will start automatically only when no other is running any longer (*). During the entire copy operations, the bandwidth seems to be a constant 7+ MB/sec, thus at least as fast as Vista Explorer, and not deteriorating over time or causing crashes. Copying 200 GB was a matter of only hours in the background and not days of exclusive copying :-)
btw: in addition to other problems, having several copy operations at the same time might also cause more fragmentation of the drive.

(*) small problem: when I started a few dozen copies at the same time, the last seems to not have seen a running process and thus started automatically, but below 20-40 parallel copies, it always worked perfectly :-)

Example 2 : parallel to the above (copy from one USB drive to second), i started copying from my emtec media cube (connected as another third USB drive :-) to my internal harddisk, and (without exactly measuring times) it looked as if both copies ran at 6+ MB/sec (instead of a Windows Explorer copy which would have split 6 or later even less MB/sec among them). btw: Windows seems to randomly assign bandwidth to those copy operations, eg sometimes splitting 6 MB/sec evenly to 2-2-2, and at other times to 1-2-3, or 1-1-4, or even 10-100-2890 KB/sec, while Teracopy seems to evenly split the available bandwidth among all running copy processes.

Example 3: this is an advanced version of example 1 ... moving 50 Files to different subdirectories on two external drives, using one of them as data storage for movies and the other as backup (with identical directory structure and contents). Not using any backup software or similar, I just copied a group of files once to one drive, and then moved the same group of files to the corresponding location on the other. Using Windows, I first need to wait for the first copy to finish until I can start moving the files (alternative: copy to both locations at the same time and manually delete files afterwards, but that once caused me a few duplicates and a few missing files since i deleted not exactly what i had started copying a few hours earlier. thus i prefer moving them). With Teracopy today, I copied the first group to the first drive (simple drag and drop in windows explorer) and immediately moved the same files to the other backup drive (simple drag and drop with shift held down in windows explorer), followed by similar copies and moves for more groups of files. I ended up with having around 40 Teracopy processes queued up on a large taskbar (alternating "copy wait" and "move wait"). After several hours 20 copies and 20 moves were properly done for a total of shuffling around 2x100 GB, and this might have been possible completely unattended instead of starting a new copy/move group every 10-30 minutes (when using windows explorer without Teracopy).

Another approach to achieve the same result could be to start only the series of copies (no moves) and set Teracopy up to not close the window on completion. After completion of a copy (or after completion of all copies half a day later), for all those finished copy windows, the destination could be changed to the other drive, the method (copy vs move) could be changed, the window set to autoclose after completion, and then the operation could be started again. Thus all copy operations would be done again for the other backup drive, but this time removing all copied files. After yet another half day of unattended work, the files would be at the intended locations.

{edit: my comp just finished 2x125+200=450 files with 2x225+100=550 GB in less than a day unattended and/or in the background; for comparison: a max bandwidth of 7 MB/sec would result in 7*3600*24/1024=590 GB in 24 hours, and standard windows explorer might slow this down to an average bandwidth below 2 MB/sec resulting in 3+ days of work around the clock with intervention at least every half hour}


In addition to the above, even the free Teracopy has some more options, a display of the list of all selected files, etc

Big Thanks again for recommending Teracopy !

6
Feature Suggestions / Re: PVD user ratings
« on: January 19, 2010, 04:32:14 pm »
I don't know whether it would be worth the effort ...
(or whether the effort would be better invested in doing some basic documentation about what menu labels really mean or how to do custom skins)

but if such a database is done anyway, it might be nice to do some additional statistics, eg counting how many people have a movie on blueray, dvd or as file (that might be another indication of how important people find some movies)

7
Support / Re: Existing Similar Entry Detected - Understanding the Options
« on: January 18, 2010, 01:20:18 pm »
1.: This line describes the entry I would like to create right now, correct?
But for what the ID number at the beginning?
3.: These are existing entries in the database.

yes, i already was wondering something similar ...

when the first line is for a "new" entry which has no ID yet, why is there an empty ID field shown on the top line ?
(maybe it is used in some other context when the same mask/dialog is shown for another purpose, but it is confusing when adding a new movie)

when the first line is for a "new" entry with a year shown, won't it make sense for the list of existing entries to also show years and maybe other info like the ID and the movie type ? (PVD doesn't get from IMDB and store whether it is a movie, videogame, etc; but at least a distinction between movies and series should be possible?). with a large part (maybe more than 1% of my database, including hidden entries) being duplicate titles (same title with different year, or a series/game as spinoff from a movie), it would be quite helpful to have that additional info in that dialog.

8
First I entered the movie title manually.
Second I download the movie data from IMDB.  This supplies the actors/actresses.  They are "hidden" by default.

At this point, the hidden entries should already have an URL leading to the correct IMDB record.

The picture you appended to your post only showed few fields (name, movie list). please go to the options and check that all checkmarks are set, including to show the URLs. Or start edit mode which shows all fields (not to change anything, but only to check whether the URL field was set)

I didn't do this yet myself for my movies and thus i said "should have an URL".
If it hasn't, there was either something wrong, or PVD couldn't get the URL (because of changes in the IMDB layout, outofdate plugins, whatever), or this would be a very important missing/broken feature in PVD (in the IMDB plugin).

At least for movies, PVD uses an existing URL instead of the title to download data for a specific movie, and people should be handled the same. The IMDB page for a movie lists the actors with links to them, and those links could/should be imported into the (hidden) people list.

I don't know how PVD (or more specific the IMDB plugin) adds new people records to the database, but it probably does this "by name" and not "by URL" since the latter might require an extensive slow search of the PVD people database. As long as only one internet database like IMDB is used, this might be no problem since IMDB always uses a unique actor name (if necessary, something like "(I)", "(XXIV)" etc is appended to the name) and in the rare cases of name changes, the URL to the people would still be the same old URL.

For movies neither such an appendix (I), (XXIV) nor the type (series, movie, game, etc) is saved, and thus there might be problems on a few people when their name changes, when there are some actors (or other professions) with the same name, etc, but that should be relatively few cases only, and even for those few cases, there should be at least one URL (if overwritten by other people with the same name) or several URLs (a list of all URLs for correct people), but never URLs for other people, and never no URL. in no case there should be 1017 people with no or multiple URLs when downloading actors with letters A-C for 713 movies.

anyway, back to the important fact in this reply, as said above:
At this point, the hidden entries should already have at least one URL leading to an IMDB record.

Quote
Third I click on the actor/actress I want to have the data on.
Fourth, I download the actor/actress info from IMDB.   In some cases there are similar names so it does not know who to pick so it shows multiple choices.

if there already is an URL (from step 2, see above), it should directly "download by URL" only that specific record and never have to get a list of "actors by name".


What I'd love to see happen is where if there are multiple names, perhaps it could choose the correct one based upon the movie(s) that are already associated with that actor/actress in my database.

Doing this search later would be very complicated, but the URL should already have been saved when the people record was created from getting the movie data, making it unnecessary to look for the movies again.

9
Support / Re: IMDB plugin shows strange behaviour
« on: November 30, 2009, 01:55:26 am »
Plugin versions can be found in "Tools -> Preferences -> Plugins"

I had looked everywhere (including this location) for half an hour and didn't find it. After your post i found it in 15 seconds when i took a look at the horizontal scrollbar :-)

since my monitor is HD+ (1920x1200), i now have enlarged the preferences window to avoid any such scrollbars, and PVD even remembers that new size on the next run. VERY nice !

There will be an improved update checking functionality available in the next version and I will place a menu item for it under "Help"

TY 1k times :-)

10
Support / Re: Problems with the IMDB People plugin
« on: November 30, 2009, 01:46:40 am »
I suspect it necessarily involves a significant change to the database structure, along with how the program handles these relationships. In other words, something to consider for the next major version, not a bug fix.

*IF* there are major changes to the database in some future version anyway, here is a small reminder what else could be changed :-)
the handling of series and their episodes could be improved to allow for better handling of the episodes' detail info, and better linking. for example: currently, the "connection" info (mostly "spoofs", "references" and similar) may lead to episodes with specially formatted titles to indicate series/season/episode which are stored as "main movies" in PVD instead of pointing to the proper episode in a serie's tree, thus generating duplicate entries and those special titles later (when downloading IMDB data) often are changed to the proper episode title and/or a localized title, losing any "first sight" info that they are episodes and no movies.

But as rick already said: something to consider for the next major version, not a bug fix
and a more detailed discussion (if any) should go to the suggestions forum ...

11
Support / Re: IMDB plugin shows strange behaviour
« on: November 29, 2009, 10:12:34 pm »
Check version number
do you really have the last one?

this seems to be a problem all the time ...
where can people easily see the versions of the program and of plugins,
and where can they easily check for updates ?

--> feature suggestion:
instead of "hiding" the check for updates (for the program itself) in "tools > preferences > internet connection > online updates", it would be nice to have such a check (also) at some more prominent location, eg on the menu option "help". And it also would be nice to have an easily accessible display of version numbers and a check for new versions for all the installed plugins (as already was suggested by other people).
To be in line with several other programs (but that's only my personal preference), these options could go to "help > check" and "help > versions" (or a combined list of loaded and most current version numbers), right above "help > about".

12
in addition, it would be nice to be able adding a picture (and maybe other data) to the midlevel of the hierarchy: the seasons. Since most DVD-releases of series are done as season boxes, the season-level of the hierarchy could get the different pictures of the season boxes which then (according to tjobbin's suggestion) in turn could be applied to all episodes in that season.

an alternate method for achieving a similar effect (without the need to modify handling of the midlevel elements of the hierarchy) might be to add N pictures to the main entry of a series which has N seasons, and automatically use the Nth picture for the Nth season and its episodes.

13
Talk / Re: video game database?
« on: November 05, 2009, 02:32:40 pm »
lots of videogames which are related to movies can also be found in the IMDB, including some data on their taglines, plots, release, genre, programmers, producers, publishers, (voice) actors, etc. Thus at least a partial database of videogames should be obtainable with PVD and the IMDB plugin.

to see/test it yourself, go to some popular movies like Alien (1979) and follow the "connection" links for "spinoff" on its movieconnections page to get to these videogames (indicated by "(VG)"): Alien (1982) (VG), Alien vs. Predator (1993) (VG), Alien Trilogy (1996) (VG), Aliens vs. Predator (1999) (VG), Aliens vs. Predator 2 (2001) (VG), Aliens vs. Predator 2: Primal Hunt (2002) (VG)

14
Support / Re: Won't let go of first IMDB selection
« on: October 18, 2009, 09:15:42 am »
It seems on earlier versions I could add an entry and paste in the url I got from IMDB.  Now it does nothing.
If I create a new entry using the '+' sign, enter a url, there's no way for it to fill in.  I seem to remember this is how I used to do it.

As I have learned in the meantime :-) ... When you want to add a new record yourself, you have to finish filling the fields (editing it) by using "apply changes", and it will only store this new record if you have added at least a title too, not only the URL. If you have selected to overwrite titles (in the plugin configuration), the value of this temporary title doesn't matter (as long as it is not matched to another/existing title), but it has to be there in the first place.

I'm still hoping that in some future version, I will be able to add movies by just giving an URL (in new movie master or with a rightclick in the taskbar to add movie from clipboard), but until then, I will have to add "x" or something as title when i add an URL to a new record, or create a CSV list with URLs and such dummynames as x1, x2, etc to add a batch of movies fast.

15
Scripts and Templates / Re: Export to csv, with "seen" field
« on: October 13, 2009, 08:46:51 am »
slightly modified from the title intro (*) of Raumpatrouille:

"What looks like a problem today might become easier in the future"
"What looks like an error today might become reality in the future"

... or just some low priority feature suggestions


(*): what sounded like a fairy tale yesterday might become reality tomorrow.
these are the phantastic voyages of the Enterprise adventures of the Orion and its crew ...

16
Support / big thanks, and a small problem with all replacements
« on: October 13, 2009, 08:19:46 am »
I will improve the plugin to support escaping quote characters in the next version.

big thanks !
with those changes, the existing features (like hyperlinks in memos) will be much more useful too !

just to be sure, since english is not our first language:
"the plugin" = import plugin? for handling escaped quote chars?

already now, using replace=""<->""" and ...;"{%value=description}";... in the export template should be simple and good enough for escaping quote chars, after another general small problem of replacements is solved:

i just tested it and currently you seem to apply each replacement twice ...
  • replace="o<->aeiou" generates Kaeiaeiouunga from Konga
  • replace="oo<->o" generates Four from Foooour
  • replace=""<->""" doubles a single " twice for a final """" instead of ""

17
Feature Suggestions / options for booleans in CSV files
« on: October 13, 2009, 06:12:27 am »

from a recent thread and some past similar ones, I got an idea for some future improvements which will probably go somewhere in the lower parts of any todo list only, but which still might be nice :-)


the first idea is about small changes to the default CSV export templates

To output boolean fields like viewed and bookmarked, include the line boolvals="yes<->no" in the options section of the template.

it is nice to be able to set the strings for output which are used for true and false. but to make templates work even if people forget (or don't know) to set this line, could this line be included by default in the templates ? It probably shouldn't be much work to do, and also wouldn't hurt if not needed ...

I also have seen that some templates have a delimiter at the end of lines. is that really needed for PVD's CSV files? usually the delimiter is not really a delimiter of all fields, but only a separator between fields, and adding a delimiter at the end causes an additional field (with empty value) to be added as last field of the record.



a second nice (more difficult to implement and also not very important) feature might be to do on purpose what someone falsly has done: setting the strings for true and false for a specific field instead of having only one definition for all booleans in a CSV file. the definition in the options section still would serve as default for those fields without their own definition.

currently, i have to use (among other options)
Code: [Select]
%OPTIONS%
boolvals="true<->false"
%OPTIONS%
%HEAD%onebool;anotherbool;defaultbool
%HEAD%{%value=wish};{%value=viewed};{%value=mybool custom="true"}
to generate something like
Code: [Select]
onebool;anotherbool;defaultbool
false;true;true
true;true;false
false;false;true

the changed syntax would allow for something like
Code: [Select]
%OPTIONS%
boolvals="yes<->no"
%OPTIONS%
%HEAD%onebool;anotherbool;defaultbool
%HEAD%{%value=wish boolvals="wish<->owned"};{%value=viewed boolvals="seen<->tosee"};{%value=mybool custom="true"}
to generate something like
Code: [Select]
onebool;anotherbool;defaultbool
owned;seen;yes
wish;seen;no
owned;tosee;yes


btw: which values are accepted as booleans by the CSV import plugin?
true/false, yes/no, 1/0, something else ?
making this also selectable probably would be overkill and pure SciFi :-)
(and thus also setting own values for different fields is even less important)

18
Support / Re: two problems with CSV-files
« on: October 12, 2009, 10:31:20 pm »

thanks for the fast reply

Quote
Was the change to UTF8BOM intended ?
Yes

in case somebody else is wondering what all this means, here is some info i found in the meantime:

after you said "yes", i read a FAQ and also twice the RFC to get a better understanding of UTF8 and what BOM does. As I understand this, the only difference caused by UTF8BOM instead of UTF8 is that a textfile (including csv files) starts with three special chars, to announce to software that the file is no simple ascii file. These three chars are hex EF BB BF or decimal 239 187 191, and they are the UTF8 representation of the Unicode character FEFF.

Since many programs (including Windows Notepad) look at these chars, it's really nice to have UTF8BOM instead of UTF8 as default (from the FAQ: "A particular protocol (e.g. Microsoft conventions for .txt files) may require use of the BOM on certain Unicode data streams, such as files."). Only in few cases a file (mostly pure ascii files) needs to NOT start with the special chars of the BOM, eg Unix shell scripts which expect specific ASCII characters such as "#!" at the beginning. only then users should be aware of the possible problem and export files without BOM.

instead of switching back to UTF8, i now have changed my program to check for this start of the file too and act accordingly, also writing these codes to a file which i generate.


Quote
Quote
which encodings should be used to make CSV-files as compatible to as many other software as possible, and which encodings are available at all ?

The export plugin has only 3 options:
  • ANSI is most compatible, BUT in this case some international characters could be lost. And since eg lots of movie titles are chinese or whatever, i won't use ANSI in most cases.
  • UTF8 and UTF8BOM both take care of almost everything. they include US-ASCII completely as the first 128 chars, and the rest is also easy to handle. for the difference with and without BOM see above.

thanks again ... question/problems part 1 solved and finished :-)



anybody else interested in what i wrote as part 2, improving importing and exporting CSV by changing delimiter, quote char and replace options ?

to find other chars which might be usable as delimiters and quote chars, i counted all chars in a CSV where i exported ID, origtitle, year and my comment, and the only chars which didn't appear on this test were %@\_`|~ and the char 127 as well as all nonprintable chars 0-31 (except CR and LF). Any ideas or comments ?

19
Support / two problems with CSV-files
« on: October 12, 2009, 02:27:41 am »

I wanted to import and export info with CSV-files, and while doing so, stumbled upon two problems. Further testing finally gave me the reasons what probably happened, but now i am looking for methods what to do ...



problem 1:
i had written a program to read a CSV file and work on the values in it, but it behaved strangely. Looking at the contents of the CSV showed nothing special, but finally i looked at every byte with my own program, and thus i found that in front of the first character were three more chars with the codes 239, 187 and 191. Notepad simply seems to ignore them, but other software might not.

when i had a closer look at the original CSV export template in PVD v14, I saw this:
Code: [Select]
%OPTIONS%
encoding="UTF8BOM"
replace=";<->,"
replace=""<->"
replace="#13#10<-> "
replace="#13<-> "
replace="#10<-> "
filter="CSV Files|*.csv"
%OPTIONS%
%HEAD%Title;Original Title;Year;Genre;Country;Actors;Director;Description;
%HEAD%{%value=title};{%value=origtitle};{%value=year};{%value=genre};{%value=country};{%value=actors};{%value=directors};"{%value=description}";

and that already was the solution to removing the three extra chars: i changed encoding="UTF8BOM" to encoding="UTF8" (without BOM) and everything works fine now.

now the questions:
  • Was the change to UTF8BOM intended ? in other/older versions and forum posts, i only had seen UTF8 and UTF-8 (which of the two is correct?), but never saw UTF8BOM ?
  • which encodings should be used to make CSV-files as compatible to as many other software as possible ?
  • and which encodings are available at all (does this depend on PVD or my system) ?



problem 2:
after importing my collection via CSV without obvious problems, i had to see that there was one problem: many comments were truncated in the middle. a closer look revealed that that happened always at some quotation marks which occured in the middle of a field.

Now the reason for the problem is obvious: PVD only imports fields "as is", respecting the userdefined delimiter and the userdefined quotation character, but only by ignoring the first and second occurence of the quotation character and the remaining string after the second quote char (assuming they enclose the field at the start and end of the field), but PVD does not interpret doubled quotation marks which are used by many other programs and even listed in a RFC.

This is no bug since the RFC and other sources give a warning: "While there are various specifications and implementations for the CSV format, there is no formal specification in existence, which allows for a wide variety of interpretations of CSV files.". Nonetheless, it would be nice if a future version of PVD would also understand those additional conventions.

Similar also applies to exporting to a CSV file: in the options section all characters which might cause problems are replaced, eg quotes are simply removed, linebreaks are replaced by space, and semicolons are replaced by commas. I found that problematic since it messes up lots of my comments, and nice features of PVD like the imbedded hyperlinks in memo fields don't work any longer when exporting and reimporting them since they will be missing the quote chars. Most serious will be the rare cases where a title is changed because it contains the delimiter or the quote char, since title and year are used to identify a movie.

now the questions:
  • would other people also be interested in more fully using import and export of CSV-files, or do most people use excel import because of this (but as i understood, exporting to excel is done with the same "normal" CSV-files and thus also the related problems) ?
  • did someone already come up with suggestions or even a solution how to change a few parameters to be able to import and export everything with as little loss as possible ?



here are some links for reference: Wikipedia and RFC 4180

and a short summary of these sources what most implementations understand:
  • fields of one record are on one line and are separated by one specific delimiter (originally "," but now often ";" since there are lots of "," and much fewer ";" in strings and most of all in numbers)
  • the delimiter separates values and doesn't terminate them (thus no delimiter is required after the last field; a final delimiter would add an empty field to the line/record)
  • any field may be enclosed with specific quotation characters
  • if a field contains the delimiter, the quotation char or linebreaks, it must be enclosed in those quote chars, and quote chars in the field are doubled to escape them
  • depending on implementation, whitespace at the start and end of fields should not be trimmed (to be sure, the whole field can be enclosed in quote chars)
  • a single header line is optional, but if present should hold the same number of fields as all the other lines, and the values should be the names of the fields

my first idea for a temporary solution for export only: changing the replace command in the options section to double quote chars instead of removing them and to no longer change ";" into "," and no longer remove linebreaks, and putting quote chars around all fields in the body of the template, or at least around all fields which at some time might include the delimiter or the quote chars, including titles, original titles, and all other strings and memo fields. this might work, but only as long as the data is not intended for reimport.

my first idea for a temporary solution for import and export: using a delimiter which is guaranteed to appear in no fields like maybe a TAB or other nonprintable char (if such chars are possible at all), but maybe someone uses TAB in comment fields. thus which char to use ? with a carefully selected delimiter, maybe quote chars wouldn't be needed at all and thus require no measures (except removing the replace commands in the header). And which second special char could be used as quote char, eg to avoid misinterpretations by PVD and other software when such software should attempt to match a userdefined quote char, which (when empty) might match any place in a string.

would anyone be interested in exploring this further, or even does have some working solution ?

20
Feature Suggestions / Re: What PVD is missing is...
« on: October 11, 2009, 08:33:10 am »
... filters ... They are what they are—simple, consistent, mathematical realities. Their menu captions have absolutely no influence over what the program does. If you don't like a default caption, change it.

((ranting or not, decide yourself))
Shouldn't labels (including the names of menu option) be an indication of what they are used for? Of course, *you* are a long time user of PVD and have much more detailed knowledge about some of its workings, and thus *you* know what a filter does/checks, and thus *you* can easily change defaults by editing some translation.
But do you expect every user of PVD to try and test what a menu option does, explore how to do such translations, and then rename that option to something which indicates what it really does instead of what it is labeled on installing the program with the default settings? Then why doesn't nostra simply name all filters A, B, C, since you can change it anyway to suit your needs?

This time, i didn't start the thread, it was not a bug report, no request for immediate action, and this is the forum for feature suggestions. Thus i find it the perfect place to suggest that some misleading standard label is changed. simply changing a label is only a matter of changing a string and not even a real change in the program. of course, it is one small additional point in some todo list, but if something like the inconsistent labels wish/owned already caused several posts, would it be so bad to avoid confusion by making it consistent all over the program, so that everybody (and most of all relatively new users) would know what is meant, being able to use it more easily? so why are you arguing so strongly?

it is good if you can change everything, but IMHO it is bad when every user has to rename something (either as some translation, or at least in his mind remembering the different meaning of some labels)

from some other posts, we can see that you had problems with those names yourself, and that you find some labels poorly chosen. here are some references for this:
PVD has a filter dedicated to "ownership," but off-hand, I don't recall how it works.
Thanks for backing me up when I lose my mind. I had disabled my custom language file in which I've named the filter "Seen/available ~ Wish list," and then forgot what "Owned ~ Not owned" meant. :-[
hehe, that is the situation for most users who don't have customized language files.
Contrary to what the caption implies, this filter is simply triggered by ....

do you really want to explain that to every new user, or would it be better to change the caption so that the implications are easy/correct ?


Quote
You can argue there should be more or different attributes supported. But we've had that discussion before, and I believe the conclusion was there is no compelling need.

Please read what i had written in my post, and you see that i agree.
my only wishes about filters were that they are named according to what they do and that (as others also have suggested, including you) advanced searches could be stored, edited and recalled in some future version and in some way or another:
I will not add a new checkbox field (unless there is smth very important to solve with it).
true and reasonable
...
Quote
You use custom fields to add any number of checkboxes and achieve functionality you need. Advanced search can be used to filter by those fields.
yes, that is exactly what i do now, but the more custom fields and corresponding different advanced searches i use, the more important it will be in the future to store and recall (or whatever other method you might have for such a functionality) those searches for quick access and filtering.
...
If you prefer a Xmas gift of substance, wish for the ability to save advanced searches to a search menu. You would then have full-blown, fully customizable "filtering" at your fingertips, instead of just one more wimpy girly filter. 8)
sounds nice, maybe even having all the userdefined searches (which really are filters, aren't they?) as an additional item in the filters menu, with two buttons on the "advanced search" dialog to "store this search as new filter" and to "overwrite an existing filter with this search" (for editing a search)


next point ...

Quote
If you have some difficulty applying or adapting the program's design to your particular circumstances, it doesn't necessarily mean there's any deficiency in the program design. The deficiency can just as easily be said to be in your personal database management/workflow design.

If you have no difficulty applying or adapting the program's design to your particular circumstances, it doesn't necessarily mean there's no deficiency in the program design. Quite often, the fact itself that there are many people who first have to adapt the program to something or ask about details of its workings instead of being able to immediately use it "as is" is a deficiency. if many people have the same problem, it might be caused by something in the program which possibly could be improved. at least not every problem automatically should be called a deficiency in the user's personal database management/workflow design.

Quote
In case you should think otherwise, I'm not joking or being dismissive about your confusion over the filter menu names. There's no requirement the fields those attributes are based on be used for exactly the same purpose the default menu captions and field names suggest.
... doesn't have to be used for that purpose. I use it to ...
So the default captions are not necessarily meaningful, and judging the program behaviour based on them is pointless.

you are mixing up something:
true, there is no requirement to use fields as the default labels suggest, but why do we have default labels at all? default labels should never indicate something misleading, but what the default action (eg "test filepath for NULL") or the default purpose is (eg "file exists"), so that every user easily can use it, and IF you use them differently, it is your responsibilty to care for the difference between name and function (by renaming, remembering, whatever).

It is also important that probably most users are no computer specialists, database managers, etc, and they should be given every opportunity to easily use the program in a way that seems to be obvious, eg suggested by some descriptions or labels, and without the need to adapt the program to something different first or having to learn about internal workings by reading in forums for hours. This is even more true when there is no manual which explains something so that you simply can say "RTFM" :-)

i am not suggesting that a manual is missing for PVD since (after some basic info) most functions are relatively obvious, but i would find it nice if also less obvious things would be changed to make them more obvious, especially if this requires no program change but only some different words on a label.



Quote
stomping on peoples ideas (whether good or bad) stifles debate and leads to people not participating

or to reply, trying to lift them up again so that they are not forgotten in some stomped hole.
oh well, and while stomping and lifting them up, so much noise is produced that it might cause the same negative effects ...

I enjoy problem solving and finding creative ways to make processes work better.
....
Given the nature of my interest, I tend to see little difference between, or reason to favour, solutions involving program changes over those involving user adaptation to the program. I may even get some perverse satisfaction from devising a workaround to what might be considered an obvious program bug or design flaw. So if someone identifies a problem and seeks a program change as a solution, I'm likely to suggest an "adaptation" if I see one.

All this is fine with me, and i agree to the fullest ... IF you don't consider your "adaption" to be the final solution to everything including real bugs. else it would be really "perverse" :-)

I think that on problem reports, it is first priority to find what exactly the problem is, and whether it is only "not knowing how to do something" (a typical "help" case), or "not being able to do something" because of a missing feature (a typical feature suggestion; btw: on a free software like PVD, there can't be feature requests :-) or whether it is a real (big or small) bug which should be fixed sometime sooner or later.
If this is determined, we all are very grateful for any hints or solutions which tell us how something can be achieved while the feature is not (yet) implemented, or what can be done as temporary fix or workaround while a bug is not yet fixed. I often find those hints very enlightening on some program details and on how other things can be achieved, and i will never oppose a suggestion for workarounds, as long as those workarounds are not said to be the solution to a real bug and that the bug need not to be fixed since there would be a workaround.

Quote
More importantly (and I think this is lost on some), there are other users reading the exchange—maybe out of specific interest in the issue at hand, maybe out of general interest about how the program works and how they might best use it. With that in mind, I usually like to put an issue in context by explaining how the program can still be used effectively despite the issue.

more support on that from me. I am eager to read such hints ...
as long as it's not labeled "since there is a workaround, there is no issue"

Quote
Without this, some posts are potentially misleading (i.e., readers who don't know what I know may think the program is broken and unusable—at least in some aspect), while others create the general impression the program is buggy.

every larger software is partially buggy or has shortcomings which could/should be improved. the difference is that some big companies fix bugs after years only (i still find some really bad "features" and "shortcomings" in Vista which i already had found in Win95/98), while nostra usually fixes important bugs in hours, other bugs also in time, and does feature improvements and new versions frequently. thanks!

even if some feature (like autocompletion :-) temporarily can't be fully used, the working "core" of PVD still is the best available. it would only be fatal if such a feature would be declared "fine as is", creating the impression that nobody cares for solutions.

Quote
But the subject matter is a good illustration of what I'm talking about. There's nothing wrong with the program's filter feature. .....

ok, let's illustrate :-)
after you explained it (!), there really is nothing wrong with the filter feature (its implementation), and neither with the rest of the above quote which i shortened ("....") and which gave some insights. The only part which was wrong was the impression which several users got from the labels of some filters (files etc), and that even more users didn't see the proper connections for other filters (wish/owned). therefore, in my opinion, we should not stop after knowing a (for many people complicated) workaround but discuss how this aspect could be improved, and since you seemed to agree on the reason ("Contrary to what the caption implies", "forgot what Owned meant when personal language file was disabled") i would hope for a simple feature suggestion (with your support) to simply rename some filter labels and settle the problem permanently for standard default installations, also for future new users.



Quote
it is ok to disagree with ideas, but given your standing in forum it seems to stop participation rather than encourage. you dont have to respond to every post
Point well taken. I often consider encouraging discussion by keeping my mouth shut, but it seems I can seldom quite bring myself to do it. Or the deafening silence that results overwhelms me.  ;)

no, please do not keep your mouth shut. I am not at all against you making suggestions how to solve something, giving hints, how to work around something etc. With the insight you give in such hints, a discussion often can be better because people get more ideas or have a better understanding of what the program does or why.

It is only a matter how you present such an advice: whether you describe your info as a hint how something can be done differently or how your info can be used as a workaround or that something might not have high priority, or whether you cancel a discussion by saying (what i called "authoritative answer", or CAD calls "with your standing in forum"; several people already have mistaken you as one of the developers) that something is no bug or that a suggestion would be pointless since no improvement would be needed when doing it like you suggest.

Pages: [1] 2 3
anything