Personal Video Database

English => Support => Topic started by: Dadeo on June 10, 2008, 04:13:39 pm

Title: Scan folders/file scanner confusion
Post by: Dadeo on June 10, 2008, 04:13:39 pm
I admit I am new to the program, but I am at a loss how to use the Scan folder/file function properly.

In the case of TV Series, sometimes the scanner puts all the file paths for all the episode files into the file path field for the existing Series record instead of putting one file path in each episode record.  Is that what it is supposed to do?  Sometimes it puts them in separate records for each Episode but then it does not group them by series! The only thing I can see is that it depends on how the files are named. If the filenames do not have spaces, they seem to all go in the one record (and it takes forever!). If they have spaces, then they go in separate records, but never in episode records (this is much faster)!

In the case of Movies with more than one file, it tries to create a separate movie record for each file, instead of putting both files in the File Path. Again this may be because there are spaces in the filename.  I don't know!

Please help!
Title: Re: Scan folders/file scanner confusion
Post by: nostra on June 11, 2008, 12:33:23 pm
Quote
In the case of TV Series, sometimes the scanner puts all the file paths for all the episode files into the file path field for the existing Series record instead of putting one file path in each episode record.  Is that what it is supposed to do?  Sometimes it puts them in separate records for each Episode but then it does not group them by series! The only thing I can see is that it depends on how the files are named.

The file scanner does not fully support episodes right now, so it will put all the file to the main movie record in the most cases. The scanner has a complex algorithm for detecting where to put files and how to combine them, but it is certainly not perfect. I plan full episodes support in file scanner and some additional scan options for future versions.

How do you name your files? Could you post some examples that produce different results?
Title: Re: Scan folders/file scanner confusion
Post by: Dadeo on June 12, 2008, 10:08:13 pm
Usually I name my files as follows:

...\Title\Title.S02E01.episodename.avi

File scan puts all these in the File Path of the main Series record with a vertical bar | separating each file.

However, I forgot to set the Preferences to leave the Titles Formatted as they are and now I have to change them all back manually or else File Scan thinks it is a new series (most often where the series name starts with 'The')  :-\

Some of the files are named:
\Title\Season\Title - 1x02 - Episodename - details.avi

File scan puts these into separate series records, not episodes, even if there is already a Series record with the Title name.

I hope that helps.  If you need any more detail let me know.
Title: Re: Scan folders/file scanner confusion
Post by: rick.ca on June 13, 2008, 12:22:27 am
Quote
The scanner has a complex algorithm for detecting where to put files and how to combine them, but it is certainly not perfect...

Your programming skills never cease to amaze me, but I have a hard time seeing how this ever will be perfect. I think you have the "right" to dictate to users a file naming convention to be used for files to be recognized by the program. Hopefully it would be flexible, but surely you need some degree of consistency. And users need to know what will be recognized. Some will have large collections named in a manner that doesn't comply, but it's not very difficult to rename files using a batch renaming utility (or a "home-made" batch file).

In the case of episodes, there are so many different ways they may be organized and named, it's probably easier for all if you decided how it should be done. I like Dadeo's ..\Title\Title.S02E01.episodename.avi, especially if, with that format, your routine could consistently identify titles—whether the episode files are in a separate sub-folders or not.
Title: Re: Scan folders/file scanner confusion
Post by: nostra on June 13, 2008, 11:13:32 am
Quote
Your programming skills never cease to amaze me, but I have a hard time seeing how this ever will be perfect. I think you have the "right" to dictate to users a file naming convention to be used for files to be recognized by the program. Hopefully it would be flexible, but surely you need some degree of consistency. And users need to know what will be recognized. Some will have large collections named in a manner that doesn't comply, but it's not very difficult to rename files using a batch renaming utility (or a "home-made" batch file).

You are right, it is impossible to include all naming conventions, but it is possible to enable users to setup how PVD should parse the file names.

Quote
In the case of episodes, there are so many different ways they may be organized and named, it's probably easier for all if you decided how it should be done. I like Dadeo's ..\Title\Title.S02E01.episodename.avi, especially if, with that format, your routine could consistently identify titles—whether the episode files are in a separate sub-folders or not.

This format will be certainly supported.
Title: Re: Scan folders/file scanner confusion
Post by: rick.ca on June 13, 2008, 05:52:45 pm
Quote
but it is possible to enable users to setup how PVD should parse the file names.

But then it would be my fault if it didn't work!  ;D
Title: Re: Scan folders/file scanner confusion
Post by: Dadeo on June 13, 2008, 08:16:24 pm
It may not be possible to recogiize ALL filenames, but it is possible to recognize and parse most of them.  I use a plugin in MediaPortal (which is fully free and Open Source) called MyTVSeries which is excellent.  It automatically links to my AMC catalog in XML format. You should consider an XML format for PVD that could be used by this plugin because MediaPortal is getting more widely known and used for the excellent Media Center it is.  You should get PVD known by these people.  This plugin does use SQL for the database I believe, but I don't know what the actual code the plugin is written in.

When parsing files, I use a regexp in the following format and it parses almost every file I have no matter how it is named:

^.*?\\?(?<series>[^\\$]+?)(?:s(?<season>[0-1]?\d)e(?<episode>\d\d)|(?<season>(?:[0-1]\d|(?<!\d)\d))x?(?<episode>\d\d))(?!\d)(?:[ .-]?(?:s\k<season>e?(?<episode2>\d{2}(?!\d))|\k<season>x?(?<episode2>\d{2}(?!\d))|(?<episode2>\d\d(?!\d))|E(?<episode2>\d\d))|)[ -.]*(?<title>(?![^\\]*?sample)[^\\]*?[^\\]*?)\.(?<ext>[^.]*)$

I don't know if this helps PVD in any way, but I just thought I'd throw it out there in case!

I agree it would be helpful to know what filenames PVD can parse!
Title: Re: Scan folders/file scanner confusion
Post by: Dadeo on June 15, 2008, 10:01:46 pm
OK, I have another problem/question!

In an existing database, when I have movies with two files (usually named Title\Title.ext -cd1 and Title\Title.ext -cd2) the filescan wants to change the file path to the second filename.  (In some cases it wants to add a new movie!).  Is there any way I can tell it to append the file to the existing File Path field like it does when I run filescan on a new database (e.g. Title\Title.ext -cd1|Title\Title.ext -cd2)

If not, could this feature be added (i.e. option to append filename to file path) where it matches an existing title?

If there are two file names in the File Path, will PVD play both files (I am guessing PVD uses the default player, so this likely depends on the player!)

If not, I guess I just need to manually add all the second file names to the File Path field or put them in the Ignore List?

Don't get me wrong, I think the FileScan feature is FANTASTIC. Especially when it automatically checks for new movies and enters them in my database. That's why I just want it to work more seamlessly!

I am still trying to think of a solution where the Title parsed from the Filename cannot match the Title field (usually because it contains punctuation which cannot be included in a filename such as a colon).
Title: Re: Scan folders/file scanner confusion
Post by: nostra on June 15, 2008, 10:32:16 pm
Quote
In an existing database, when I have movies with two files (usually named Title\Title.ext -cd1 and Title\Title.ext -cd2) the filescan wants to change the file path to the second filename.  (In some cases it wants to add a new movie!).  Is there any way I can tell it to append the file to the existing File Path field like it does when I run filescan on a new database (e.g. Title\Title.ext -cd1|Title\Title.ext -cd2)

No, it is not possible


I will review all the suggestion on this topic and improve the File Scanner accordingly in the next major release. Please, be patient.
Title: Re: Scan folders/file scanner confusion
Post by: Dadeo on June 16, 2008, 08:17:30 am
Thanks Nostra, don't worry I am just learning, not impatient!  I appreciate all your great and timely support and the efforts you are making!

What I did was create a PVD using file scan, then import all the results into an AMC database then import all the file paths from that AMC database into my PVD database.  I had to use the AMC import because I don't think the CSV import lets you merge or overwrite data on matching titles.  Am I right?  It would be great if the CSV import could do this as well  ;)

Just a note on a few weird problems on the filescan parsing in Titles:

1.  10,000 becomes 10,00
2.  3 10 becomes 3 0
3.  Mr. (also Mrs.) becomes Mr (Mrs) looks like all dots (periods) are stripped

These files may or may not import movie info from the web depending on the import site.  If I correct the titles, filescan keeps finding them as new movies.

Title: Re: Scan folders/file scanner confusion
Post by: nostra on June 16, 2008, 12:28:11 pm
Quote
If there are two file names in the File Path, will PVD play both files (I am guessing PVD uses the default player, so this likely depends on the player!)

PVD will play both files using a temporary .pls file. It works with most video players.
Title: Re: Scan folders/file scanner confusion
Post by: patch on June 19, 2008, 04:32:12 pm
I admit I am new to the program, but I am at a loss how to use the Scan folder/file function properly.
...
In the case of Movies with more than one file

I to are confused, and would like to know how I should name files for the scanning to work in 3 different situations.

1) Movie over more than one disc. It seams to identify "Shrek cd1.avi" "Shrek cd2.avi" etc as multiple discs for the same movie.

2) Later movies in a series I usually put in the same directory (eg "Shrek 1.avi", "Shrek 2.avi", "Shrek 3.avi"). It seams to assume it is all one movie over multiple files so combines them. I'm not sure how I should name them to prevent this. Also is there an efficient way of splitting up the enteries after they have been scanned (and falsely grouped)?

3) Movies with a sample file / trailer eg "Shrek - sample.avi", "Shrek.avi" are assumed to be two separate movies. Does Personal Video Database have a convenient way of handling these trailer files? How should I name the files so they are appropriately recognised.

Last of all. Has writing cataloguing data back to the movie files been considered, in a similar manner to ID3 tags are used for mp3 files. If .avi files do not support ID3 style tags them perhaps a separate file could be added to the directory containing the movie file, similar to how subtitles are handled with .srt files
The reason I ask is then the movie file would retain the cataloguing information enabling subsequent scanning to be much more accurate. Useful if you have your movie collection on a file server accessed by multiple computers and multiple user accounts, like I do  :)
Title: Re: Scan folders/file scanner confusion
Post by: nostra on June 19, 2008, 05:22:13 pm
Quote
1) Movie over more than one disc. It seams to identify "Shrek cd1.avi" "Shrek cd2.avi" etc as multiple discs for the same movie.

PVD identifies CD1..CDN and Title1..TitleN

Quote
2) Later movies in a series I usually put in the same directory (eg "Shrek 1.avi", "Shrek 2.avi", "Shrek 3.avi"). It seams to assume it is all one movie over multiple files so combines them. I'm not sure how I should name them to prevent this.

Episodes are not supported yet.

Quote
Also is there an efficient way of splitting up the enteries after they have been scanned (and falsely grouped)?

You can try to right click the movie and select "organize files by episodes". This function recognizes sXXeXX format.

Quote
3) Movies with a sample file / trailer eg "Shrek - sample.avi", "Shrek.avi" are assumed to be two separate movies. Does Personal Video Database have a convenient way of handling these trailer files? How should I name the files so they are appropriately recognised.

Such files are not supported yet. I'll add this to my TODO list.

Quote
Has writing cataloguing data back to the movie files been considered, in a similar manner to ID3 tags are used for mp3 files. If .avi files do not support ID3 style tags them perhaps a separate file could be added to the directory containing the movie file, similar to how subtitles are handled with .srt files
The reason I ask is then the movie file would retain the cataloguing information enabling subsequent scanning to be much more accurate. Useful if you have your movie collection on a file server accessed by multiple computers and multiple user accounts, like I do  Smiley

You can use export function to create such text files. Naming video files according to data from database is planned for future.
Title: Re: Scan folders/file scanner confusion
Post by: patch on June 20, 2008, 01:03:55 am
Thank you for you wonderfully prompt response. You have been a great assistance.

I assume from
Quote
PVD identifies CD1..CDN and Title1..TitleN
Then to ensure sequels to movies are treated as separate movies I should ensure my titles do not inadvertently follow this format, for example I should use
"Shrek (2001).avi", "Shrek II (2004)" instead of
"Shrek 1 (2001).avi", "Shrek 2 (2004)"

Quote
Movies with a sample file / trailer eg "Shrek - sample.avi", "Shrek.avi" are assumed to be two separate movies.
Quote
Such files are not supported yet. I'll add this to my TODO list.
A similar issue occurs with movies I have two copies of (high and low resolution versions).
Or separate episodes of a TV serial which PVD has not recognised.

One way to facilitate suitable file associations would be to this would be to have an extra option in "Optimise Database..." which looks for separate entries with identical imdb link and asks how they should be linked (ie main movie, blue-ray/dvd versions, trailer, episode in series, or delete duplicate file)

Quote
Has writing cataloguing data back to the movie files been considered, in a similar manner to ID3 tags are used for mp3 files.
Quote
You can use export function to create such text files. Naming video files according to data from database is planned for future.
The function I'm trying to achieve is 100% rescan accuracy of "Scan for new movies/changed paths..." allowing fully automated rescan. I suspect to get the graphic images to rescan correctly I many need more than a text file but I suppose if it contained the Amazon link a text file would do.
(100% rescan accuracy is valuable when the movie files are on a central file server, accessed by multiple computers & users within the home. Also useful if file server structure is changed over time).

So my question is, do any of the output formats write a file to the same directory as each movie file, and will PVD then use this data when subsequently rescanning the file server from a new computer / user account.
Title: Re: Scan folders/file scanner confusion
Post by: nostra on June 20, 2008, 01:13:28 am
Quote
Then to ensure sequels to movies are treated as separate movies I should ensure my titles do not inadvertently follow this format, for example I should use
"Shrek (2001).avi", "Shrek II (2004)" instead of
"Shrek 1 (2001).avi", "Shrek 2 (2004)"

Theoretically PVD will handle both version, but you can do this to be sure.
Title1..TitleN would mean:
"Shrek1 (2001).avi", "Shrek2 (2004)" without spaces


Quote
Quote
Movies with a sample file / trailer eg "Shrek - sample.avi", "Shrek.avi" are assumed to be two separate movies.
Quote
Such files are not supported yet. I'll add this to my TODO list.
A similar issue occurs with movies I have two copies of (high and low resolution versions).
Or separate episodes of a TV serial which PVD has not recognised.

One way to facilitate suitable file associations would be to this would be to have an extra option in "Optimise Database..." which looks for separate entries with identical imdb link and asks how they should be linked (ie main movie, blue-ray/dvd versions, trailer, episode in series, or delete duplicate file)

I'll consider implementing smth like this.

Quote
do any of the output formats write a file to the same directory as each movie file

No :(

Quote
and will PVD then use this data when subsequently rescanning the file server from a new computer / user account.

No :(

Could you describe your tasks and working environment more precisely, so I can imagine what changes need to be done.
I do not guarantee that all requested features will be available, but I am doing my best to suit most of the users.
Title: Re: Scan folders/file scanner confusion
Post by: patch on June 20, 2008, 02:44:41 am
Quote
Could you describe your tasks and working environment more precisely, so I can imagine what changes need to be done.

With pleasure

Movie files are stored on a file server and accessed by multiple users / computers within the home.
PVD is an excellent way of selecting a movie to watch so I would like to use it on each computer on the network.
Setting up PVD on each computer requires "Scan folders for new movies" and pointing it at the file server.
Doing this the first time is labour intensive as I need to manually check each imdb and Graphics/Amazon link.

Doing it from subsequently computers could be fully automated (which would make PVD network friendly)
Probably the easiest way to achieve this is if PVD had "*.id3" export and import formats.

Export *.id3 would write a file to the same directory as each movie on the hard disk, using an identical name but .id3 file extension.
The id3 file would contain at least the imdb link, but preferably other generic fields (eg graphics or link to Amazon, but not individual user data such as their rating & if they have watched the movie).

During "Scan folders for new movies" PVD would then look for an .id3 file. If found it could add the movie without user intervention, making the "Scan folders for new movies" 100% accurate from a new computer or user account.

In an ideal world the id3 tag information could be appended to the movie data file, similar to mp3 music file. Unfortunately I suspect .avi does not support it http://msdn.microsoft.com/en-us/library/ms779636(VS.85).aspx and http://en.wikipedia.org/wiki/Comparison_of_container_formats I believe some of the later movie file formats support Metadata /Tags, so this may happen in the future, but in the mean time a separate file would achieve the same end.
Interestingly this program appears to put similar information within .avi files http://www.exefind.com/abcavi-tag-editor-P21011.html
I thought I read somewhere that it does it by putting the information within unallocated blocks within the file but I could be wrong. Actually I don't want to distract you as an external file is probably easier and would achieve the functionality I'm after.

BTW
id3 file format is not critical, I just choose it as it is relatively standard and currently stores similar information. A .xml files would probably work just as well.

Title: Re: Scan folders/file scanner confusion
Post by: nostra on June 20, 2008, 03:05:33 am
OK, I understand.

One question: would it not be much better and easier if you install a database server on your Server PC and have a single PVD database on the server as well that is accessed by all the users in the network.

I plan to implement this in the next major update and it is much easier for me to implement than id3-like export/import.

P.S. the server can run on windows or linux. (Firebird)
Title: Re: Scan folders/file scanner confusion
Post by: rick.ca on June 20, 2008, 03:55:44 am
Quote
Has writing cataloguing data back to the movie files been considered...

When I first read this, I found the idea appealing. It would offer greater peace-of-mind PVD would not be "confused," should I move or rename movie files. I'm trying out Media Center, and one of my disappointments is I cannot import PVD data into it. Probably just as well—I don't want to be maintaining two databases. But if PVD could write back to the movie files, I suspect MC would read the information automatically. In stark contrast, I've always maintained my music information in ID3 tags, and all of it is seamlessly available in whatever software I choose to use.

I'm not familiar with the tagging convention, but a lot of meta information can be saved in AVI files. I imagine the first issue here is MediaInfo (the DLL PVD uses to get information from movie files) only reads, does not write information. So nostra would have to figure out how to do that. And what about other file formats? While writing only to AVI's for this limited purpose may not be difficult, it opens the door to what could be a major new feature that would probably not be easy to support: What about other programs using the same information? Being able to get my PVD information into MC would be great, for example, but what would PVD do when MC wrote its version of the information back to the tags?

Quote
Movie files are stored on a file server and accessed by multiple users / computers within the home...

I suppose each user would want to maintain their own database, but based on the same store of movies on the server. Nevertheless, would it not make sense to built a "master" database of all movies on the server? Then each user could start with a copy of that, and delete the movies they're not interested in.

Quote
Probably the easiest way to achieve this is if PVD had "*.id3" export and import formats.

I thought ID3 was for music only. Much of the information is the same, but does it provide for tags unique to movies? I suppose a tagging convention—if that's what it is—doesn't necessarily address the issue of how such tags are included in various file types. Thus your suggestion the information could be saved to separate files. I've never come across anything, however, that would suggest other programs could make use of information saved this way. Even if this works for your specific purpose, saving the information in the movie files removes the risk of the files getting separated.

Title: Re: Scan folders/file scanner confusion
Post by: patch on June 20, 2008, 10:07:48 am
One question: would it not be much better and easier if you install a database server on your Server PC and have a single PVD database on the server as well that is accessed by all the users in the network.
A server version would definately help with some but not all of what I'm trying to acheive.
It would allow multiple users to access the PVD on the file server  :)

The difficulties I would have are:
1) The "File server" in my case is just a PC with large hard disks which are shared over the network. People log in to multiple accounts to use it as a PC, so it would not be easy to ensure the server version of PVD was running, and running only once.

2) File storage updates. I regularly need to buy more hard disk space, which often results in the path to some of the movie directories changing. If the basic movie description data is stored with the movie file then PVD rescan is virtually garanteed to accurately update it's database (even if I needed to open a new database & scan in the file server again). With a server version I'm not as confident server configuration updates would go as smoothly.

3) When we go on holidays it is nice to bring a laptop +/- an external drive with some movies. If PVD supported tagging I could effectively bring it too, as setting it up for the new configuration would be trivial.

To illustrate how it could work, I can describe how audio is handled
I digitise my music, tag it, and put it on the file server (resulting in key track data being stored with the sound data).
Media Monkey http://en.wikipedia.org/wiki/Mediamonkey is configured to rescan the file server directories on startup.
So any new music is available to everyone at home.
As each user runs their own copy of Media monkey, they also have personalised data such as ratings, play lists, last played & frequency etc. (Some users even still persist with iTunes or Winamp)

I would like to achieve a similar thing with my movies, hopefully through PVD.

What about other programs using the same information? Being able to get my PVD information into MC would be great, for example, but what would PVD do when MC wrote its version of the information back to the tags?

I do not think movie file tagging is widely used or standardised yet (which is why we are having this discussion). As such I think it is very unlikely any tags created by PVD would automatically work with MC. However I believe tagging will be more widely used in the future, and at least a de facto standard will emerge with time.

Quote
I suppose each user would want to maintain their own database, but based on the same store of movies on the server. Nevertheless, would it not make sense to built a "master" database of all movies on the server? Then each user could start with a copy of that, and delete the movies they're not interested in.

Master databases worry me. Simple single user programs accessing movies on a file server sounds less complex and mor flexible to me.

Quote
I thought ID3 was for music only
..
saving the information in the movie files removes the risk of the files getting separated.
Agree
ID3 was suggested as a format to start from that much of the industry is familiar with.
Incorporating movie tags within movie data files will no doubt be the ultimate standardised solution. The separate file was a simple way of achieving the same functionality with all movie file types without compromising compatibility with movie players or needing to write to large video files. Yes the files would need to be copied together if movie files were moved, but I already do that with subtitled movies. Tags within movie files (.avi .mkv .wm etc) could be progressively added later as resources and tools permit.
Title: Re: Scan folders/file scanner confusion
Post by: rick.ca on June 20, 2008, 02:14:35 pm
Quote
I do not think movie file tagging is widely used or standardised yet...

That seems to be the case. I just spent hours trying to get image tag information into MC—it seems hopeless. I recall abcAVI Tag Editor saves IMDb information in AVI's, but I suspect the tags are the author's creation, and he's just writing them as a block into the file. So maybe the answer is it's easy to write information into AVI's, it won't affect other software that needs to read them, but there's little chance the information could be shared in any way.

Quote
Master databases worry me...

I was assuming this would be a one-time thing—just for getting started. I meant "master" only in the sense it would include all movies on the server, and users would start their own database using a copy of it.

I suppose your vacation scenario might be an example of where this functionality is required on an ongoing basis. But I wonder about that too. Wouldn't you just copy your database to the laptop, and then run scan folders for changed paths, pointing to the new file location? Yes, you want 100% accuracy, but in this case, the program only needs to check for changed paths—not new files. If it's trying to match the title directly to the new path while ignoring the filename already recorded, I'd say the code needs to be revised to take the old filename into account. I can't imagine why anyone would move files and change file names and expect PVD to rematch everything all at the same time. Furthermore, even though you can't drag a server with you on vacation, there's no reason not to take your entire database. This would be much easier and faster than rebuilding a new database. PVD will do a good job of telling you what's available to watch, and what you left behind.

Quote
Yes the files would need to be copied together if movie files were moved, but I already do that with subtitled movies.

I have no doubt you can do it—you're the one running an industrial network in your home! Other users don't where or how there files are stored. For them, separate files increases the risk of things going wrong. But then they're not likely to be interested in this feature, are they?
Title: Re: Scan folders/file scanner confusion
Post by: patch on June 21, 2008, 04:40:21 am
Quote
I do not think movie file tagging is widely used or standardised yet...
That seems to be the case. I just spent hours trying to get image tag information into MC—it seems hopeless. I recall abcAVI Tag Editor saves IMDb information in AVI's,

To get MC to use tag information it both needs to be written to the movie file, and MC needs the code to recognise it. I suspect you would have problems with both steps ATM.

Quote
Quote
Master databases worry me...

I was assuming this would be a one-time thing—just for getting started. I meant "master" only in the sense it would include all movies on the server, and users would start their own database using a copy of it.
I had assumed that it was proposed a server version on PVD would need to be run on the file server and client versions run on other computers or accounts. For this to work I need
1) Server implementation of VDB (Master database implementation)
2) Computers sharing movie files to be switched on
3) Server VDB to be running once somewhere
4) Client version of VDB to be run by users

My preferred set-up is to just reliably re-scan my movie collection from each computer / account. For this to work I need
1) Version of VDB with 100% accuracy on re-scanning files (after it has been done once manually & results exported)
2) Computers sharing files switched on (running operating system file sharing)
3) Stand alone version of VDB on client computer (& intermittently rescanning shared file directories)

Quote
Quote
Yes the files would need to be copied together if movie files were moved, but I already do that with subtitled movies.

I have no doubt you can do it—you're the one running an industrial network in your home! Other users don't where or how there files are stored. For them, separate files increases the risk of things going wrong.

You over estimate what I'm doing.
"File server" is a windows XP machine the "share folder" enabled the the directories containing my movies.

I find some movies have more than one file (trailer, subtitles). So for consistency I put each movie in it's own directory, so adding another file to that directory would be a trivial change. For those that have multiple movies in one folder they could either move the whole directory also or sort by file name and move the movie file & associated tag file. If not manual re-scan would be require. Either way I believe it is workable until tags within movie file are better supported.

In summary
What I'm really trying to achieve is 100% rescan accuracy without human intervention. Independent of path to the movie, user account I'm scanning from or computer I'm using. This ability can then be used for many applications including a multi-user network environment, movie storage updates, holiday set up etc.

The only way I can see this being achieved is if the manual selections made during the initial scan are stored with the movie file. The simplest implementation being a file containing the URL to the movie description (eg imdb) and cover image (eg Amazon).

The rest of the discussion is really just trying to predict what direction the industry is likely to go in the future by comparing how it is done with music files (a more mature technology).
Title: Re: Scan folders/file scanner confusion
Post by: rick.ca on June 21, 2008, 07:42:08 am
Quote
I had assumed that it was proposed a server version on PVD would need to be run...

No, no, no! I know it's confusing, but this is a three-way conversation. My recommendations are never the same as nostra's.  ;D

I understand why you would prefer not to run a server version of PVD. I was agreeing with you in recommending each user run PVD stand-alone, using their own separate database. I was merely pointing out it would not be as much work to create those separate databases as you seemed to think it would be.

Quote
The only way I can see this being achieved is if the manual selections made during the initial scan are stored with the movie file.

Again, I'm not questioning the means, I'm questioning the objective. Your scheme is only necessary in the event one needs to rebuild a database starting with nothing but the movie files. This is not necessary if you are just changing the location of files—the rescan function works with 100% accuracy (I tested it since my last post). It is not necessary if you are creating a new database that is a subset of an existing one (your vacation scenario).

I understand you want any user to be able to build a database by selecting files and letting PVD do the rest. I've already suggested your request may be easy to implement. Please don't think I'm arguing this because I don't think you should have the feature you want. Nostra is perfectly capable of weighing the pros and cons and the amount of effort required. I'm countering your implication there is something fundamentally wrong with PVD when it's actually performing extremely well—according to its intended purpose. If any further effort is required in this area, I have to wonder if it should not be directed to making the initial scan more accurate. Everyone would benefit from that.
Title: Re: Scan folders/file scanner confusion
Post by: patch on June 29, 2008, 01:27:06 pm
While I still think (in the short term) a separate file containing tag information is an efficient way of supporting all video formats and avoiding player compatibility issues.

It is interesting several programs support writing tags within movie files such as
IMDB Movie DB http://www.dancemammal.com/moviedb.htm
abcAVI Tag Editor http://www.exefind.com/abcavi-tag-editor-P21011.html

I note the movie file reader PVD uses supports reading some movie data, pity it didn't support writing as well
http://mediainfo.sourceforge.net/en
Title: Re: Scan folders/file scanner confusion
Post by: seabass on July 22, 2008, 10:43:55 pm
Quote
Master databases worry me...

First of all sorry for my bad english..
I don’t know if I my suggestion is to primitive, but wouldn’t it be the easiest way to solve all the problems if you would create a simple importfilter which can access a normal pvd databasefile then you just have to include somewhere in the preferences the option to enter a filepath to the central pvd file on the server and choose this plugin for standard import and you would always have 100% accuracy.

..or if a pvd file cant be accessed on a server because of the restrictions of firebird just put an exported csv file and a folder with the pictures there.
Title: Re: Scan folders/file scanner confusion
Post by: rick.ca on July 23, 2008, 05:50:19 am
I'm not sure I understand your idea, but it gives me one... The ability to export to, and import from another PVD database might be a useful "utility" feature. It would provide a convenient means for creating a database that is a subset of another, and to share records with other databases. I'm assuming the export of all movie information, including all images and associated people. It probably should include the option to exclude "personal" information like date seen, rating, comments, etc. The advantage of this over a CSV/XLS export/import is the resulting movie records would not need to be updated.

Neither idea satisfies patch's need. If I understand correctly, he has multiple users running independent versions of PVD, but sharing a common movie collection. Your feature or mine could be used to establish a new database for a new user, but won't help on an ongoing basis. A problem remains with updating new movies (i.e., downloading information). Some movies cannot be identified by scripts without some intervention by the user. Once this has been done by one user, it would be helpful if the identifying information (e.g., IMDb record number, suitable poster URL, etc.) was automatically available to the other users—so their updates would be fully automatic.

Nice ideas, but I think the priority should be the improvement of the program's ability to correctly identify movies from file names. If were successful 99% of the time, there may be no point in adding more features to deal with the 1%.
Title: Re: Scan folders/file scanner confusion
Post by: nostra on July 23, 2008, 02:00:35 pm
There will be a possibility to append another PVD database and copy/paste records from one database to another in 0.9.9.

For the patсh's problem: it is already possible to write a script that would generate a txt file with any information needed for each movie placing it near video file. Then a script like this http://www.videodb.info/forum_en/index.php?topic=904.0 could be used to retrieve the information from txt files.
Title: Re: Scan folders/file scanner confusion
Post by: rick.ca on July 24, 2008, 05:09:26 am
Quote
There will be a possibility to append another PVD database and copy/paste records from one database to another in 0.9.9.

Just to clarify... The append function would include all information, including custom fields not in the destination database? Will there be the option of excluding "personal" information like date seen, rating, comments, etc? If I wanted to share a subset of my database with another, I would delete unwanted records from a copy of my database, and then the copy would be appended to the other.

I presume copy/paste would only include the record copied, not any associated records—so the pasted records would need to be updated.

Quote
For the path's problem...

Let me outline this to see it I understand correctly...


So, if a new movie had been previously updated (by another user in a shared collection environment), the information necessary to ensure a fully automatic update would be found in the text files. If not previously updated, a text file would be created after the update was successfully completed.

It's not clear to me if this will work for posters as well. If a value (a URL, I presume) for a poster is read from the text file and saved to the database, will the existing poster plugins use this information to download the poster?

In the circumstances, this seems to be the most practical solution. All we need are some kind volunteers to write those scripts.  ;)
Title: Re: Scan folders/file scanner confusion
Post by: patch on July 31, 2008, 11:43:26 am
For the patсh's problem: it is already possible to write a script that would generate a txt file with any information needed for each movie placing it near video file. Then a script like this http://www.videodb.info/forum_en/index.php?topic=904.0 could be used to retrieve the information from txt files.

Excellent.
This should achieve the functionality I was looking for.
Thanks for implementing it.
Title: Re: Scan folders/file scanner confusion
Post by: patch on August 01, 2008, 07:23:10 am
a script like this http://www.videodb.info/forum_en/index.php?topic=904.0 could be used to retrieve the information from txt files.

Might be some value in extending this to extract the imdb link out of .nfo if present and not previously defined in database or tag file (.txt)
Title: Re: Scan folders/file scanner confusion
Post by: nostra on August 02, 2008, 10:15:21 pm
I do not have time now for improving the scripts. You can do this if you wish...
Title: Re: Scan folders/file scanner confusion
Post by: patch on August 19, 2008, 10:55:07 am
For the patсh's problem: it is already possible to write a script that would generate a txt file with any information needed for each movie placing it near video file. Then a script like this http://www.videodb.info/forum_en/index.php?topic=904.0 could be used to retrieve the information from txt files.

Has anyone had any luck with this?

I had a look at scripts and they all seam to relate to importing data into PVD, typically via a url reference, parsing html page, and filling relevant database fields.

Plugin templates seam more relevant to me. The closest I could find is the "Web Pages(Index&Movies)" template which produces one (html) file per movie.
I'm guessing what is need is to:

1) Set "recordfilename" to an absolute address based on "File Path" rather than an address relative to the index file. Not sure how to handle multiple entries in path, probably need to process them the same as PVD does when grouping files, or txtinfo_en.psf http://www.videodb.info/forum_en/index.php?topic=904.0 does in function GetDownloadURL : String; Had a go at the latter but couldn't get it working :(

2) Produce a pages/movie.ptm to export the text file contents, probably based on "XML (Movies).ptm". Looks like it wouldn't be too bad to adapt to what txtinfo_en.psf wants

3) Run the export function to write tags to all known movie files.

4) Produce a txtinfo_en+imdb+allmovie.batch to import movie data into new user accounts. Maybe straight forward but I don't really understand how the various importing options relate to Scripts and Plugins
Title: Re: Scan folders/file scanner confusion
Post by: patch on August 20, 2008, 11:49:37 am
Plugin templates seam more relevant to me. The closest I could find is the "Web Pages(Index&Movies)" template which produces one (html) file per movie.
I'm guessing what is need is to:

1) Set "recordfilename" to an absolute address based on "File Path" rather than an address relative to the index file. Not sure how to handle multiple entries in path, probably need to process them the same as PVD does when grouping files, or txtinfo_en.psf http://www.videodb.info/forum_en/index.php?topic=904.0 does in function GetDownloadURL : String; Had a go at the latter but couldn't get it working :(

I now suspect this is not possible.
Looking at http://www.videodb.info/forum_en/index.php?topic=766.0 and experimenting with "Templates for plane text export" I suspect templates for export plugin do not support functions and recordfilename does not support absolute addressing (sub-files must go in the same directory as user selects to save the index).

Looks like I will have to hope nostra adds tagging in 0.9.9 or learn to write my own export script.
Title: Re: Scan folders/file scanner confusion
Post by: nostra on August 20, 2008, 11:55:51 am
Quote
Looking at http://www.videodb.info/forum_en/index.php?topic=766.0 and experimenting with "Templates for plane text export" I suspect templates for export plugin do not support functions and recordfilename does not support absolute addressing (sub-files must go in the same directory as user selects to save the index).

Yes, it is not possible with a standard export plugin, but can be solved pretty easily with a script. I have no problem with writing a script for you, but only after 0.9.9...
Title: Re: Scan folders/file scanner confusion
Post by: patch on August 20, 2008, 12:25:06 pm
I have no problem with writing a script for you

That is great news as with txtinfo_en.psf that will implement movie tagging functionality

but only after 0.9.9...

Sounds like sensible prioritisation to me. I'm looking forward to seeing 0.9.9 also :)
Title: Re: Scan folders/file scanner confusion
Post by: patch on October 29, 2008, 02:21:39 pm
Sounds like sensible prioritisation to me. I'm looking forward to seeing 0.9.9 also :)
This is frustrating.
Just finished backing up my movies, updated PVD data then copied it to the backup drive.
Or that was what was suppose to happen. Unfortunately I rushed the last step and actually copied the old copy over the new one.

Now I'm back to on old copy of my PVD data which cannot be opened by a non-administrator. So will need to export to CVS, manually import pictures, & redo last 2 months cataloguing.

I'm really really looking forward to being able to export tags. Doing this stuff twice because of my own mistakes is very frustrating.
Title: Re: Scan folders/file scanner confusion
Post by: rick.ca on October 29, 2008, 07:35:34 pm
Ouch! Sorry to hear that. :'(

You don't want to hear this, but for my own sake I need to repeat my mantra: "All my backup routines shall be automated, multi-versioned and otherwise idiot-proof. The biggest risk to my data is ME!"
Title: Re: Scan folders/file scanner confusion
Post by: patch on October 29, 2008, 09:39:36 pm
The biggest risk to my data is ME!"
:)
Yep, I agree. Was just a bit annoyed with myself last night. Oh well, today is another day.
Title: Re: Scan folders/file scanner confusion
Post by: patch on December 04, 2008, 01:15:38 pm
How do you name your files? Could you post some examples that produce different results?
Usually I name my files as follows:
...\Title\Title.S02E01.episodename.avi

Some of the files are named:
\Title\Season\Title - 1x02 - Episodename - details.avi
Same however naming is not as consistent as it should be. All TV series files have season & episode indicated but it varies for example season 1 episode 2 may be specified by any of the following
s1e02
s01e02
01x02
1x02
[01x02]
[1x02]
102
In most cases this numbering is proceeded by the series name however the punctuation & captilisation varies. In some the series name is omitted altogether. In some a "series description" follows the numbering, in others just encoding information.

What is consistent is I put all files pertaining to a series in the same directory.
...\Title\s1\
...\Title\s01\
...\Title\Season 1\
Less commonly
...\Title\

IMO
For PVD to have any real chance of picking up all these files it would have to assume once one episode of a TV show has been found in a directory, then the other files in the same directory tree also pertain to the same TV show. This would enable much less stringent matching criterion without confusion from movies (which may spans multiple CD or DVD). Not unlike how vob & similar files from a dvd rip are handled.

This would make adding episode to and existing TV series in PVD much more reliable.
How to respond when PVD first encounters a news TV series is more difficult, as after one is confirmed then other files become additions to an existing series. Might be easiest it users just add one episode first then re-run the file scanner to add all other episodes (unless someone can think of a more elegant solution).
Title: Re: Scan folders/file scanner confusion
Post by: rick.ca on December 04, 2008, 01:49:11 pm
Quote
For PVD to have any real chance of picking up all these files it would have to assume once one episode of a TV show has been found in a directory, then the other files in the same directory tree also pertain to the same TV show.

I wouldn't be very happy with that. Most of my episodes go into one directory until viewed, then deleted. If it's too difficult to code a scanner that can understand a variety of different file naming conventions, I'd rather one be imposed. Considering any of the possible sequences of characters that denote season and episode are very unlikely to mean anything else, would it not suffice that the title appear first, followed by the season/episode, and then anything after that is ignored?
Title: Re: Scan folders/file scanner confusion
Post by: nostra on December 04, 2008, 01:51:28 pm
I think I can manage to make the scanner understand multiple naming formats, not 100% precision but still pretty precise.
Just stay tuned for the next beta ;)
Title: Re: Scan folders/file scanner confusion
Post by: patch on December 06, 2008, 01:31:42 pm
From 0.9.9.x Beta --> 0.9.9.3 New plugin system
Why would I spend time manually preparing files in such a way that I wouldn't have to use the built-in automatic functions? It is also on the edge of the scope of a file scanner to retrieve info not from the file itself but from an array of files paired with the video file.
I do not believe anyone intends to manually generate such files.
You are correct PVD current internet search capability is the best way of collecting such information.
But there use is the same as id3 tags for music files.
As such they are best written by an export command from within PVD.
They could then be read by any other instance of PVD (eg another user on the same computer or same network scanning a shared directory, or movies copied to a completely separate network). Once movie descriptions are stored near movie files they can also be used to transfer information to other programs (the same as is currently done with music files). It is correct that writing tag information to internal tags would be preferable, but that is not supported by mediainfo.sourceforge.net and unlikely to be supported within the foreseeable future see http://sourceforge.net/forum/forum.php?thread_id=1695277&forum_id=297609
Title: Re: Scan folders/file scanner confusion
Post by: blue334 on December 08, 2008, 11:02:49 am
I think what you're asking can largely be acheived using the current export plugin.
What you're describing is essentially an advanced html export template.

Where you lose me is where you suggest this kind of information storage should be read by PVD directly. PVD is after all a database app, you cannot acheive anything like the current functionality outside the database engine. A browser would be better suited.
Title: Re: Scan folders/file scanner confusion
Post by: patch on December 08, 2008, 12:50:37 pm
I think what you're asking can largely be acheived using the current export plugin.
What you're describing is essentially an advanced html export template.
When I last tried (0.9.8.20), the export function went close. It was able to create 1 file per media entry with useful data in it. However what I could not achieve is to put this file near the media file (same directory & name as media file but different file extension) as the path variable wanted an absolute address. I didn't try exporting poster images the same location but with a jpg extension, but believe that to would be useful.

So yes I believe I am after an enhancement to the export function.

Where you lose me is where you suggest this kind of information storage should be read by PVD directly. PVD is after all a database app, you cannot acheive anything like the current functionality outside the database engine. A browser would be better suited.
It's value is in transferring information between instances of PVD or to another similar application.
The ways I intend to use it is
1) I store my multimedia an shared computer drives. When I add movie to the collection I scan it into PVD, check the display name, link to imdb, and cover are correct.
2) I would like to be able to write this information back to the movie (or it's directory)
3) There are many other media users in this household who access the movies from the multiple user accounts on the multiple computers. To avoid having them redo step 1, PVD could just scan and read the data written in step 2 (it already has the ability to scan a directory on start up, and read information from a text file near the movie file).
4) Similarly if some of the movies were copied to another server, where different users ran PVD, then they too could scan the identifying data into their copy of PVD automatically.
5) I believe media centre is also likely to be able to read tags of this sort, enabling efficient transfer of information between applications

This is the same functionality achieved by tagging mp3 files, and basically has the same advantages.
Title: Re: Scan folders/file scanner confusion
Post by: nostra on December 08, 2008, 01:03:21 pm
The export plugin will be improved with an extra option to save files near video files in the next version.
I am also planning an export template and some additional import functionality (including file scanner) using predefined XML files near video files.
Title: Re: Scan folders/file scanner confusion
Post by: patch on June 14, 2009, 01:30:46 am
Tag export/import (like ID3 for mp3) (http://www.videodb.info/forum_en/index.php?topic=1193.0") has been marked as already added. This is good news if that is the case. Looking at 0.9.9.10 I can't see where it is implemented.
Further to discussions here (http://www.videodb.info/forum_en/index.php?topic=1249.msg5576#msg5576)

Actually I'm not sure what is going to be the most efficient path to getting movie tagging functionality.
Importing & exporting, text & image data near a movie file is part of the functionality required.
However I suspect working reliably with the file scanners name parsing and multi-part file identification could well be more challenging.

To explain
For a single file movie the tag information would probably go in .jpeg and .txt (or .xml or .pvd?) files with identical path and base file name.
For multi-file movies the base file name is less obvious as it would involve stripping out the disc number or using the first file name or exporting for all component files
For DVD images (ie with .vob files) the tag files could go within the directory containing the .vob files or 1 level above; with a generic file name or directory name (if 1 level above .vob files)
For TV series similar issues arise however then there is the need to identify the episode, season, series, as well as grouping with other episodes in the same series.

As such it wouldn't surprise me if nostra intentionally left it till other parts of the file scanner were implemented & stable.

mmm
Reading through this again, it looks as if I'm making it all too hard so perhaps it is time to stand back & think what it is all about.
My aim is for PVD to import media with maximum reliability and minimum user intervention.
The problem is PVD needs to make a connection between the video data I have & various internet databases but PVD can't watch movies and it is often ambiguous which record in and internet database is preferred. So an indirect link needs to be used. The best way of doing this is going to vary with the information available to PVD, for example:

1) Analyse file (& path) name to deduce movie name, then use that to look up internet databases. PVD does this very well (& flexibly with regex). The only enhancement I can think of is sometimes the enclosing directory has a better name, so being able to easily select that would be handy.

2) Some media files have an associated .nfo file containing a link to an internet database (most often imdb). It would be good if PVD could automatically extract the reference. I'm actually not sure if it is doing this already as it has been discussed. Knowing what .nfo file to associate with what media file could be a problem as they are generated by humans, thus done inconsistently. The way to start is support the easy cases ie if have .nfo with same base file name as media or folder with one .nfo & one media file then PVD could assume they belong together. Ideally PVD would display some how that it found the reference in the "Scan results" dialogue box. Perhaps the action could show "Add movie" imdb link. This would save the user worrying about the information in the file name.

3) If the movie has already been imported into PVD, and links to internet databases checked, then it would be good if that information could be written back to the movie file, so if it was moved to another system, it could then be imported into PVD there with minimal user intervention. Writing data back to the movie file is also likely to assist communication between media programs (is it does for music with mp3 tags). For PVD though the value is to provide assistance to the file scanner, so the writing protocol needs to be linked to the file scanner reading protocol. Again just doing the easiest case (single movie file) would be useful (provided it cleanly ignored the more complicated case and its functionality was documented eg in change log)
Title: Re: Scan folders/file scanner confusion
Post by: rick.ca on June 14, 2009, 07:22:20 am
Quote
Tag export/import (like ID3 for mp3) has been marked as already added. This is good news if that is the case. Looking at 0.9.9.10 I can't see where it is implemented.

I marked that off because I did not interpret the feature request to necessarily be an integrated tag management system, and I thought most of the pieces were in place for this to be done on an ad hoc basis. Perhaps I'm wrong, but it doesn't seem you've actually tried to do anything with the tools that have been provided. I'm not trying to give you a hard time, but since I don't really have much interest in this, I was expecting you would give it a shot.

Is there not an template command for exporting text file beside a media file? That could be used to export a text file with the URL. I assume the Text File plugin can import that. Those two could export/import any text, but, as I understand it, the URL is your primary concern—with that, another instance of PVD would not have to search IMDb to find the movie. I believe there's also a template command for exporting posters beside media files. And there's Image by ID to reassociate them with a new database. That would work only for new databases, so I do understand your interest in Reset's plugin.

Quote
The only enhancement I can think of is sometimes the enclosing directory has a better name, so being able to easily select that would be handy.

You should be able to handle that with regex as well, unless the pattern is hopelessly inconsistent. What is the source of the directory and file name?

Quote
Some media files have an associated .nfo file containing a link to an internet database (most often imdb).

I don't see much point in this. Sure, it there's a reliable URL consistently available in a consistent format, then this may help, but the primary approach is supposed to be to get the filename (and maybe year) and use that to search IMDb. In normal circumstances, that works fairly well. What if the URL is wrong? In that case, a mess would have to be undone, and the normal method repeated—after removing the offending NFO. Maybe what would be more practical would be to import the entire contents of any NFO into a custom field. Then, if any of the information was needed, it would be readily accessible.
Title: Re: Scan folders/file scanner confusion
Post by: patch on June 14, 2009, 09:37:20 am
Quote
The only enhancement I can think of is sometimes the enclosing directory has a better name, so being able to easily select that would be handy.

You should be able to handle that with regex as well, unless the pattern is hopelessly inconsistent. What is the source of the directory and file name?

Media file names are often very cryptic eg "vmt-agmihtf-xvid.avi" but the enclosing directory is more informative "A Good Man Is Hard to Find". Maybe others don't have this experience, but I find it is rather common. Currently I cut & paste in the "Scan results" dialogue to make the name more informative for the imdb lookup. This is a wish list suggestion to improve to the "Scan results" dialogue to make this process more efficient. Regex would not make the manual editing more efficient as the very reason it is a manual process is the naming is generated by humans so is inconsistent.

Quote
Some media files have an associated .nfo file containing a link to an internet database (most often imdb).

I don't see much point in this. Sure, it there's a reliable URL consistently available in a consistent format, then this may help, but the primary approach is supposed to be to get the filename (and maybe year) and use that to search IMDb.

URL are in a consistent format (they are a valid url). If one is available it will always point to the correct imdb record with greater accuracy than a search (duplicate records in imdb, similar movie titles etc). So if it is available it is worth using, if not then falling back to file naming & database search is of course required. Currently I can "Open enclosing folder" open the .nfo file, cut & paste from there. Importing into a custom field would provide similar functionality but not address why I suggested it ie the improve scanner performance.

Note I have spent time making file names look pretty in the past, but given I now access my movies from PVD the actual file names and paths are largely irrelevant. If I need to change them to import into PVD then that indicates the efficiency of operating PVD could be improved.

.nfo are supplied with at least 50% downloaded content and most have imdb link, so I suspect I'm not the only one who would benefit from this functionality, hence it's suggestion as a scanner enhancement wish list item

Quote
Tag export/import (like ID3 for mp3) has been marked as already added. This is good news if that is the case. Looking at 0.9.9.10 I can't see where it is implemented.

I marked that off because I did not interpret the feature request to necessarily be an integrated tag management system, and I thought most of the pieces were in place for this to be done on an ad hoc basis.

PVD is run on an open database with various import & export capabilities. It has always been possible to import movie information from various sources in an adhoc fashion. The aim of using tags is to further improve the file scanner performance. To do so it becomes part of the file scanner. Maybe PVD has all the tagging capability nostra wanted to implement, if so then a some what adhoc solution must be the answer.

BTW
The above suggestion are in-order of solution specificity, from most general (but least exact) to most exact / potentially automatic.
Title: Re: Scan folders/file scanner confusion
Post by: rick.ca on June 14, 2009, 12:05:42 pm
Quote
Note I have spent time making file names look pretty in the past, but given I now access my movies from PVD the actual file names and paths are largely irrelevant. If I need to change them to import into PVD then that indicates the efficiency of operating PVD could be improved.

Obviously, the actual file names and paths are relevant. A lot of work went into a sophisticated file scanner that can accurately extract a title and year—which in turn will very likely result in a successful IMDb search. All that is required it that there be reasonable degree of consistency in the file path names. The fact that you're unwilling to use it as designed doesn't mean it's efficiency needs to be improved.

But fine. You're right, there are probably others who expected PVD to accurately process inconsistent, meaningless and misleading file path names used by many torrents. Most of the torrents that provide reliable NFOs are also more likely to use reasonable names, but let me be generous: Let's say that 50% of these torrent file names are so bad you really need a URL from an NFO. But only 50% of those have NFOs. So that means the you're still going to get an unacceptably high error rate.

I use "clean" file names, and the file scanner performs flawlessly. My file manager includes a regex file renamer. One button cleans movies, another episodes. Maybe I'll configure a third to deal with those that have a more accurate title in the containing folder. Even with some requiring manual adjustment, it still works out to seconds per file. And then the filenames are accurate and meaningful in my file system as well as in PVD. The same thing can be done with a number of excellent freeware file renamers.

Quote
The aim of using tags is to further improve the file scanner performance.

Maybe now that the scanner works so well with clean filenames, we don't need tags. Maybe what would be much more useful would be the ability to rename files. Files could be renamed to Title (Year) IMDb#.ext and a variable added so regex could capture the IMDb# and add the URL.
Title: Re: Scan folders/file scanner confusion
Post by: patch on June 14, 2009, 02:56:49 pm
My file manager includes a regex file renamer. One button cleans movies, another episodes. Maybe I'll configure a third to deal with those that have a more accurate title in the containing folder. Even with some requiring manual adjustment, it still works out to seconds per file. And then the filenames are accurate and meaningful in my file system as well as in PVD. The same thing can be done with a number of excellent freeware file renamers.

That sound like a good system.
I got sick of doing it by hand. The issue I found was often multi-part files were used and for replay all files names needed to match (sub-titles, multiple discs etc). As such it was easier to leave the cryptic but matching set of file names. Typically the folder names are more human readable, hence my approach.

Does the file manager you use cope with multi-part files appropriately? Which one do you use?
You do of course realise there is a certain amount of similarity between what you are doing with your file manager and what I (& I assume others) are trying to do with PVD file scanner.

But only 50% of those have NFOs

imo PVD being able to link directly to the exact imdb reference would be a very useful feature. Currently I do this by hand which is much less efficient.
True if I knew the Exact imdb title & year the text search would work well, but often I do not know this prior to looking it up on imdb.

Maybe what would be much more useful would be the ability to rename files. Files could be renamed to Title (Year) IMDb#.ext and a variable added so regex could capture the IMDb# and add the URL.

That is a neat idea.
We would need to ensure it worked with multi-part files.
I would like at least the option to append the old file name ending to the new cleaned name, maybe Title (Year) IMDb# [old descriptors].ext as I'm not sure I would really like to loose some of that information (CAM, TS, dvdrip, HDTV, BluRay)
Not sure if it would cope with graphics data but given the often reasonable image PVD can pull from imdb I still think it is a good approach.
Title: Re: Scan folders/file scanner confusion
Post by: rick.ca on June 14, 2009, 08:11:36 pm
Quote
Does the file manager you use cope with multi-part files appropriately? Which one do you use?

Directory Opus. It's renamer only does what the regex I specify does (it also has other modes, like find & replace, and script—where file attributes and tags could be used). I haven't run into problems with multi-part files. When I do, maybe I'll be inspired to modify the regex.

Quote
You do of course realise there is a certain amount of similarity between what you are doing with your file manager and what I (& I assume others) are trying to do with PVD file scanner.

Yes, I thought about that and concluded that just because a similar function is required at two different stages of my workflow does it mean it has to be restricted to one or the other. It does mean that something has to be done at the first stage. You don't have to be very sophisticated at the file renaming stage—just change it so the essential variables (title, year, season, episode, disc#) are in a fixed pattern. Then the second stage regex can't miss—they're just pulling out those variables from a fixed pattern. Also, I should emphasize the renaming stage is not automatic, but under my supervision. So I'm also using my own judgment as to what is required. That may range from tweaking the regex and running it on 100 new files, to deciding the one file name I'm processing right now is easier to edit manually.

BTW, all this is explained in 0.9.9.x File Scanner and Regular Expressions (http://www.videodb.info/forum_en/index.php?topic=1168.msg5356#msg5356). This post includes useful regex references and a link to a good freeware renamer.

Quote
True if I knew the Exact imdb title & year the text search would work well, but often I do not know this prior to looking it up on imdb.

Torrent file names often include a year (which is easy to identify with regex). It seems to me that even if the title is inexact, including the year dramatically improves the IMDb search result.

Quote
I would like at least the option to append the old file name ending to the new cleaned name, maybe Title (Year) IMDb# [old descriptors].ext as I'm not sure I would really like to loose some of that information (CAM, TS, dvdrip, HDTV, BluRay)

Just specify the regex that will do so. Instead of skipping over them (or, for example, dropping everything after finding the title and year), put those parts in variables and append them as [old descriptors].

Quote
Not sure if it would cope with graphics data but given the often reasonable image PVD can pull from imdb I still think it is a good approach.

Don't forget it's now very easy to export posters to images "beside" the media file. So if a lot of effort has gone into getting better quality posters from elsewhere, this is a reasonable thing to do—to make those available to other PVD databases and/or just to be a visual reference in the file system.

A nice companion to a file renaming feature would be the option to save the poster beside the file at the same time.

Nostra, I'm warming up to the idea of starting a 1.0 wish list (http://www.videodb.info/forum_en/index.php?topic=908.msg5558#msg5558). It seems I now have a number of feature requests that could go with my recommendation for Scanners (1981) (http://www.imdb.com/title/tt0081455/). ;)
Title: Re: Scan folders/file scanner confusion
Post by: patch on June 15, 2009, 04:14:27 am
Quote
You do of course realise there is a certain amount of similarity between what you are doing with your file manager and what I (& I assume others) are trying to do with PVD file scanner.

Yes, I thought about that and concluded that just because a similar function is required at two different stages of my workflow does it mean it has to be restricted to one or the other.

mmm
PVD already has regex, it already stores user defined regex expressions for the file scanner.
As a wish list item I would like to combine the essential features of rick.ca file renamer into PVD. To explain:-
The "Scan results" dialogue should show the fields it will potentially use to search the online databases & set up the record. For movies these are
Title
Original title
Year
url
Path

To that I would like added the ability to run user defined regex (via buttons or menu selection). Typically they would take the "path" as the input string & output to the above fields as appropriate. To be general it maybe worth having tha ability to start from other fields. I imagine some may prefer to run a regex on the existing Title to clean it.
It would also be useful to make the url active so the linked page could be viewed in the users default browser.

For series a similar thing could be done, but more fields are required
Season
Episode
Episode title

The user needs also to be able to correct the scanner if it gets it wrong. A possible solution is to have a series check box, which when ticked indicates the video is part of a series and the extra fields are shown (Unless it is felt the scanner would always be correct). I have not thought through how to incorporate the add to series (http://www.videodb.info/forum_en/index.php?topic=1283.0) functionality, but that too would be applicable here.

The reason I would prefer it as part of PVD is I like the way PVD file scanner just shows me the new or changed files when I point it a at directory trees. I also think a dedicated tool like PVD could do a better job of managing video files.

The actual file name updating could then be done as a second step, useful for those who look at the file names or want to ensure subsequent accurate re-scanning.

Quote
I would like at least the option to append the old file name ending to the new cleaned name, maybe Title (Year) IMDb# [old descriptors].ext as I'm not sure I would really like to loose some of that information (CAM, TS, dvdrip, HDTV, BluRay)

Just specify the regex that will do so. Instead of skipping over them (or, for example, dropping everything after finding the title and year), put those parts in variables and append them as [old descriptors].

Sorry I didn't explain myself, I was thinking how I would like a PVD file renamer to work, not how Directory Opus works

Quote
Not sure if it would cope with graphics data but given the often reasonable image PVD can pull from imdb I still think it is a good approach.

Don't forget it's now very easy to export posters to images "beside" the media file. So if a lot of effort has gone into getting better quality posters from elsewhere, this is a reasonable thing to do—to make those available to other PVD databases and/or just to be a visual reference in the file system.

A nice companion to a file renaming feature would be the option to save the poster beside the file at the same time.

Good ideas
Title: Re: Scan folders/file scanner confusion
Post by: rick.ca on June 15, 2009, 09:30:23 am
Quote
The reason I would prefer it as part of PVD is I like the way PVD file scanner just shows me the new or changed files when I point it a at directory trees. I also think a dedicated tool like PVD could do a better job of managing video files.

I don't think there's any advantage in trying to turn PVD into a file manager. Don't be mislead by the fact I use regex in my file manager. Those regex can do no more or less than those in the scanner configuration. The difference is, if there anything that needs to be done manually, the file manager is the appropriate environment in which to do it. If I had a good eye and didn't care about the file names, I could just make the bare minimum of changes necessary so the scanner regex would work. As a practical matter, renaming the files in the file manager not only "cleans" them, but makes it obvious which ones need manual adjustment so that the scanner regex will work.

The function of the scan results dialog should be restricted to correcting associations for movies already in the database, or enabling a more accurate search. In other words, it's for moving forward to the next step, not for fixing errors from the previous step. In a well designed work flow, this dialog should more often than not just be a simple confirmation.

Quote
Sorry I didn't explain myself, I was thinking how I would like a PVD file renamer to work, not how Directory Opus works

A PVD renamer would be rename files based on user-configurable database variables like Title, Year, IMDb#, etc. To use such "other information," an additional regex variable would have to be provided that you could use to save the data in a custom field. Then, of course, the renamer would have to allow variables for custom fields. But allowing any custom field to be used in a file name doesn't seem like a good idea. Maybe the Features field could be used for this purpose.
Title: Re: Scan folders/file scanner confusion
Post by: nostra on June 17, 2009, 12:45:45 am
OK, I will be laconic:

1. I do not think there will be some built-in support for tagging. I only think of realizing this functionality with scripts/plugins

2. There will be file rename feature, but only to organize files for existing movies in database

3. I am very curious about your 1.0 feature suggestions, Rick
Title: Re: Scan folders/file scanner confusion
Post by: rick.ca on June 17, 2009, 04:17:15 am
Quote
3. I am very curious about your 1.0 feature suggestions, Rick

I haven't thought about it much, other than in this conversation. But to summarize:

1. The scan results dialog should allow the user to make changes...

a) For records already in the database, forcing the match in cases where the scanner "missed" and thinks a file is new. This would include forcing a match of an episode to an existing series when the scanner has failed to recognize it as an episode.

b) For records not in the database, changes to title and year, to increase the likelihood of a successful web search.

2. As noted here (http://www.videodb.info/forum_en/index.php?topic=1283.msg5555#msg5555), the scanner should not identify files as series unless a season and episode is determined by the regex.

3. The introduction of a regex variable for storing an online database code from which a URL can be constructed from base URL mask saved in the user's configuration. For example, an IMDb mask could be http://www.imdb.com/title/tt[URL code]/, where [URL code] is the 7-digit IMDb title code.

4. A file path renaming utility which uses relevant database fields as variables. This sort of thing can be rather dangerous unless the user can see the results before committing—it needs some kind of "old path ~ new path" preview, or perhaps some kind of integration with a reintroduced Gridview (where a preview can be shown in a column beside the existing file path). Examples of what should be possible:

[parent]\[title] ([year]) [rating] [url code].[ext]
[parent]\[title] ([year])\{DVD rip files}
[parent]\[series] S[s#] E[e#].[ext]
[parent]\[series]\Season [s#]\[e#] [episode title].[ext]

5. Since some (like patch) will use a renaming utility to ensure other users/databases will be able to get online information with 100% accuracy, it would be nice to also save the current poster using the same name—at the same time the renaming is done (i.e., as an option, "Save poster with same pathname?").

I think a file renamer should have the effect of making users much more accepting of the scanning process. They only need to change new filenames to get results that are "good enough" (that depends on the user—how good their regex are and how willing they are to make corrections in the results dialog). Once the "processing" is done, they will be able to use the renamer to make the file pathnames "perfect."
Title: Re: Scan folders/file scanner confusion
Post by: Nebol on October 04, 2009, 03:42:05 pm
In the case of episodes, there are so many different ways they may be organized and named, it's probably easier for all if you decided how it should be done. I like Dadeo's ..\Title\Title.S02E01.episodename.avi, especially if, with that format, your routine could consistently identify titles—whether the episode files are in a separate sub-folders or not.

Problem is, that would not work at all for me, since, in my system, I definitely can not have periods in the file names, only before the suffix. I use underscores. Title_S02E01_whatever.suffix
Also, using periods like that is damn ugly.
Title: Re: Scan folders/file scanner confusion
Post by: nostra on October 04, 2009, 04:29:03 pm
What system are you using that would not let you use periods in file names???
Title: Re: Scan folders/file scanner confusion
Post by: rick.ca on October 04, 2009, 06:24:55 pm
The periods are irrelevant. They could also be underscores, spaces or pipes. The point is there just needs to be a consistent structure to the filenames so the regex can work.
Title: Re: Scan folders/file scanner confusion
Post by: patch on October 05, 2009, 01:08:58 am
The point is there just needs to be a consistent structure to the filenames so the regex can work.
We could probably do that by listing (default) supported formats in the wiki.
Currently this aspect is avoided
http://www.nimidia.com/pvd_wiki/tiki-index.php?page=Organize-files-by-episode
http://www.nimidia.com/pvd_wiki/tiki-index.php?page=Adding-a-New-Movie&structure=PVD-Manual

But I may have missed it as I'm having problems using the search function in the wiki.
Title: Re: Scan folders/file scanner confusion
Post by: rick.ca on October 05, 2009, 04:17:26 am
Quote
We could probably do that by listing (default) supported formats in the wiki.
Currently this aspect is avoided

That would just encourage the misconception the program is intended to handle any filenaming convention one cares to use. I'd rather just say a consistent filenaming convention like Title (Year).ext for movies and SeriesTitle S00 E00 [EpisodeTitle].ext for series will be foolproof. The regex provided will handle many forms in addition to these. The regex can even handle a somewhat inconsistent filenaming convention, but that gets complicated very quickly. Considerable care has to be taken to ensure the regex do not conflict with one another, and are applied in the correct order.
 
In fact, a list of "supported formats" would be misleading unless it also enumerated all the possible conflicts. This recent post (http://www.videodb.info/forum_en/index.php?topic=1502.msg6597#msg6597) is a good example of the difficulty. You can name movie files Year - Title [(Director]).ext if you like, but then you have to provide a regex that will recognize this pattern (including the Director part, whether or not it exists) and remove (or reorder) any regex that would misinterpret the year as an episode number.
Title: Re: Scan folders/file scanner confusion
Post by: patch on October 05, 2009, 06:26:06 am
Quote
We could probably do that by listing (default) supported formats in the wiki.
Currently this aspect is avoided

That would just encourage the misconception the program is intended to handle any file naming convention one cares to use.
Not if it was listed as

PVD is designed to support...
I can in fact extract the required information if you use other naming conventions but you may need to manually edit the title during "scan folders for new movies / changed file paths" or modify the regex expressions in "Tools -> preferences -> File scanner". See ... for further details


My intention is we would document the default / preferred naming. Illustrating some of the flexibility in punctuation & capitalisation, but not covering every possibility as that is best done by reading the regex search terms.
Sorry I didn't make that clear.
Title: Re: Scan folders/file scanner confusion
Post by: rick.ca on October 05, 2009, 09:13:25 am
I don't think it's a good idea to suggest there are "default" or "supported" formats by way of example. The regex are the most concise definition of what is supported. True, most people don't understand regex. But the best way to deal with that is to provide an example for each expression and "walk through" exactly how the pattern recognition works. Then it would be clear what file names will be recognized, how much flexibility there is, and possibly what modifications would be required to handle different or multiple patterns.
Title: Re: Scan folders/file scanner confusion
Post by: patch on October 05, 2009, 10:19:35 am
I don't think it's a good idea to suggest there are "default" or "supported" formats by way of example.

np
I will leave it to your judgement then how best to direct none technical users to naming conventions which work with PVD
Title: Re: Scan folders/file scanner confusion
Post by: rick.ca on October 05, 2009, 07:48:28 pm
Thanks. I look forward to your post or wiki entry on the matter.