Personal Video Database

English => Support => Topic started by: buah on March 04, 2010, 04:23:44 pm

Title: Using NULL in Advanced Search
Post by: buah on March 04, 2010, 04:23:44 pm
EDIT: The original name of this topic was Funny Thing, as I can remember. Obviously the things became more serious here ;)

Funny thing.

Filters -> Viewed = 2533

1. + Advanced search->rating IS NULL =1817, or
2. + Advanced search->rating=0 =1817, or
3. + Advanced search->rating IS NOT NULL =2533?

Heh? :D

And I thought I learned something about logic
Title: Re: Using NULL in Advanced Search
Post by: rick.ca on March 04, 2010, 07:47:58 pm
Did this not help?

Quote
NULL Fields are always treated a bit differently by the database engine.

See Expressions Involving NULL in this reference (http://www.firebirdsql.org/manual/qsg2-databases.html#qsg2-databases-sql). I've found the tip of thinking of NULL as UNKNOWN particularly helpful.
Title: Re: Using NULL in Advanced Search
Post by: buah on March 04, 2010, 11:21:30 pm
Did this not help?

Quote
NULL Fields are always treated a bit differently by the database engine.

See Expressions Involving NULL in this reference (http://www.firebirdsql.org/manual/qsg2-databases.html#qsg2-databases-sql). I've found the tip of thinking of NULL as UNKNOWN particularly helpful.

I thought so Rick. But when it says that all 2533 is not uknown (IS NOT NULL) and at the same time that of those 2533, 1817 is uknown (IS NULL) I don't know what to think anymore...
Title: Re: Using NULL in Advanced Search
Post by: rick.ca on March 05, 2010, 12:26:13 am
If IS NOT NULL is obtained by NOT-ing IS NULL and NULL means "unknown," then NULL might be expected to identify empty fields, but it's meaning is still unknown, so NOT-ing the result quite obviously doesn't exclude them, and now everything IS NOT NULL...

You might find it easier to do a simple search for Rating = 0. ;D
Title: Re: Using NULL in Advanced Search
Post by: buah on March 05, 2010, 08:08:15 am
You might find it easier to do a simple search for Rating = 0. ;D

It's getting harder thant to import 6000 movies ;)

When I do search rating=0, for the results I got also entries which rating field is empty, not 0? That's how all began, look at my first post
Title: Re: Using NULL in Advanced Search
Post by: rick.ca on March 05, 2010, 12:33:00 pm
My position on using "0" as a rating for really really bad movies instead of the more obvious meaning of "unrated" hasn't changed...

It can only be "bad" if you choose to look at it that way. So "0" means "unrated." Most would agree, any movie rated less than 5 is "bad." You can't possibly argue a scale from "1" meaning the worst movie of all time to "5" is insufficient to rate the nuances between bad movies. You can search for "rating <3 AND >0." Clicking on the extreme left-hand side of the stars graphic removes (or resets to 0, if you prefer) the rating.

Also, considering most professional reviewers use only four or five stars, we should be able to get by with ten. ;)
Title: Re: Using NULL in Advanced Search
Post by: buah on March 05, 2010, 04:39:09 pm
Ok, if I understood you well, and please correct me if not so, I need here to clarify some things.

I'm in the last phase of importing and updating all my movies to PVD - rating seen movies. So, I filtered movies to seen ones and that's how I got 2533 entries. Some of those 2533 seen movies are rated, and some not.

Of those 2533 how to get a list of only not rated movies?

The only logical answer for me was to apply IS NULL search because their rating field should be empty (uknown). That's how I got 1817 entries.

How to get a list of only rated movies?

I reset advanced search filter. The only logical answer for me was to apply IS NOT NULL search because their rating field should not be empty (filled in with some numbers, including zeros. That's how I got 2533 entries??? But how? It was supposed to be -1817 entries that aren't rated? Weren't the result should been 2533-1817=716?

Edit: None of my movies ever had rating "0", I just tested search against it.


Edit: corrected counting mistake, thx mgp
Title: Re: Using NULL in Advanced Search
Post by: mgpw4me@yahoo.com on March 05, 2010, 06:00:54 pm
Weren't the result should been 2533-1817=1716?   716

If you know that some fields have ratings, then you can use:
(rating is null or rating < 1) for unrated movies
(rating is not null and rating > 0) for rated movies.
Title: Re: Using NULL in Advanced Search
Post by: buah on March 05, 2010, 06:35:06 pm
Weren't the result should been 2533-1817=1716?   716

If you know that some fields have ratings, then you can use:
(rating is null or rating < 1) for unrated movies
(rating is not null and rating > 0) for rated movies.


1. Why it isn't possible to get desired result only with NULL search?
2. Tnx, but nope mgp, your tips don't work in my case, still same results... (1817 and 2533)

[attachment deleted by admin]
Title: Re: Using NULL in Advanced Search
Post by: AimHere on March 05, 2010, 08:00:56 pm
Did a little quick testing on this issue myself... what I've found is that "Rating = 0" and "Rating IS NULL" are equivalent for the purposes of Advanced Search.

I have 1364 movies in my database, and either of the above search criteria returns 1329 visible records. A search for "Rating > 0", on the other hand, returns 35 movies... 1329 + 35 = 1364. (Clearly I have some work to do regarding rating my movies, hehe.  ;D)

When editing a movie record that has any non-zero rating, you can actually enter "0" in the text box next to the stars and save the record... and the movie essentially becomes "unrated". (If you edit the movie record again, the text box will actually be blank now, but blank and "0" are still equivalent.) Same thing happens in edit mode when you click to the left of the leftmost star, or click-and-drag on the stars and move your mouse off the left end before releasing; the movie becomes unrated.

So, the search really boils down to:


Then, just be sure to use a rating of at least 0.5 (which may as well mean "absolute excrement"  ;D) when rating all your movies, and you're set. Never use a rating of "0" except to clear out the rating altogether.

What threw off your results using mgpw4me's suggestion was his inclusion of "rating is not null" in the second line.

Side note: I think the reason why "Rating IS NOT NULL" returns all records might be due to PVD actually assigning a value (in this case, zero) to the rating of EVERY record at the time of creation, and maintaining a value in that field at all times. A value of "zero" (meaning unrated) simply isn't shown visibly in the text box next to the stars. So, all records have some value for "Rating", and hence, the "Rating" field is never NULL.

Aimhere
Title: Re: Using NULL in Advanced Search
Post by: mgpw4me@yahoo.com on March 05, 2010, 08:05:46 pm
I just checked my database and my method works correctly with imdbrating (395 not rated, 3168 total, 2773 rated)...NULL check should not be necessary.  I have a funny feeling that when you edit a movie, the rating might change from NULL to '0', which may account for your situation.

You'll have to add checks for each of the other ratings that is being used.
imdbrating > 0 or rating > 0 or additional rating > 0  (any movie that has a rating)
imdbrating < 1 and rating < 1 and additional rating < 1 (any movie with no rating)

AimHere is right...as best I can tell.
Title: Re: Using NULL in Advanced Search
Post by: AimHere on March 05, 2010, 08:08:29 pm
Okay, I've done a little more testing, and I think I've figured out how the "NULL" and "NOT NULL" conditions really work.

"NULL" means that a field currently has no value (as you might expect), which is useful for searching text fields. So you could, for example, search for "Description IS NULL", get a list of all movies which have no descriptions, and fill them in.

"NOT NULL", on the other hand, seems to mean that a field has had data in it at some point, regardless of whether or not it currently has data.

As a test, I ran two advanced searches: "Tagline IS NULL", which returned 1263 records out of 1364 movies, while "Tagline IS NOT NULL" returned 107... but 1263 + 107 = 1370, not 1364! The six-movie difference can be explained by the fact that six movies preciously had values in the "Tagline" field, which were subsequently cleared out. Since they're currently blank, they are included in the "NULL" condition... but they had data in them at least once, so they are ALSO included in the "NOT NULL" condition!!! In other words, "NULL" and "NOT NULL" are NOT mutually exclusive.

If I pick a random movie which has never had a tagline and give it one, then re-run both searches, the counts change to 1262 and 108. I then re-edit the movie's record, clear out the tagline, save, and run the searches once more... 1263 and 108! So, while the "NULL" result might fluctuate up or down as fields are given data or cleared out, the "NOT NULL" results will only ever increase with normal program usage.

Given this counter-intuitive behavior, I'd recommend avoiding the use of the "NOT NULL" condition for anything. A real database guru might be able to provide a valid reason for this logic, but for most people it's totally useless.

Aimhere
Title: Re: Using NULL in Advanced Search
Post by: AimHere on March 05, 2010, 08:11:54 pm

imdbrating > 0 or rating > 0 or additional rating > 0  (any movie that has a rating)
imdbrating < 1 and rating < 1 and additional rating < 1 (any movie with no rating)

Remember, a movie can actually have a fractional rating of 0.5, so your second line won't catch them all. You'd actually need:

imdbrating = 0 and rating = 0 and additional rating = 0


Aimhere
Title: Re: Using NULL in Advanced Search
Post by: mgpw4me@yahoo.com on March 05, 2010, 08:21:19 pm
Forgot about the floating point field.  I'm not fond of assuming that '0' will always be there, so I'd still check against < .01

NULL means 'unknown value'.  Given the PVD interface, I don't see anyway you can set a NULL value.  Simply clearing a field won't make it null...you have to run an SQL update command to set the value to null.

Reference material for those who care:
http://www.databasedesign-resource.com/null-values-in-a-database.html
Title: Re: Using NULL in Advanced Search
Post by: rick.ca on March 05, 2010, 10:26:12 pm
I hope we got this figured out without anyone's head exploding! ;D

AimHere's observation is a bit disconcerting. It's strange to see a text field classified as NULL and NOT NULL at the same time. In the case of a text field, using "0" instead of "NULL" is not available as a means of avoiding the anomaly. :-\
Title: Re: Using NULL in Advanced Search
Post by: buah on March 05, 2010, 10:40:10 pm
Hey guys,

First of all, thank you for your time as well as for been interested to help me.
Second of all thank you for preserving my common sense, whether or not aware of it.
What might happened here, according what you wrote Aim, is that I imported "rating" xls column to PVD in the time of creation my .pvd.

Well, please stay with me now.

"Rating" xls column had some cells without values ("not rated"). But, were those cells considered to be NOT NULL? Yes, in my opinion! How?

How did I get "Rating" xls column? By exporting CATVids field "My rating" to xls. In CATVids, there were movies that were unrated. Were those records considered to be NOT NULL?
No, in my opinion, they were NULL because (condition of) those records were generated in CATVids, but not imported to it! CATVids (.mdb) database was unaware of them.

So, what happened? When you export information that some record IS NULL (no value, "not rated") to xls cell,
for that cell that information that something (what IS NULL) is imported into it, is what makes it NOT NULL!!! Regardless of xls cell becomes empty, Excel became aware of it's NULLness. So to say: it became known (NOT NULL) that it is uknown (NULL)? NOT NULL wins here.

What happened while exported further from Excel to PVD? The very known (NOT NULL) value was imported to PVD - "empty"! NOT NULL wins again!

When NULL wins? When you generate completely new record within PVD. At the moment "rating" is empty and is truly NULL. Aim, you were totally right about this: You enter some value, than erase it - it's not NULL any more: it became empty with the value "empty" (so NOT NULL)

So my final conclusion on NOT NULL would be:
NULLing is transferable only between databases. Otherwise, you can never export/import NULL information (condition) to preserve it as NULL. As such it can only exits inside the database.

Please, confirm what I think or discuss it further.

If we agreed on this, than what is left to conclude is that what was caused this topic and made a mess, and what was the real question here is (look at my first post again):

How the hack is possible that "Rating=0" = "Rating IS NULL"?

Any thoughts?


EDIT:
It's strange to see a text field classified as NULL and NOT NULL at the same time.
Rick, you posted this while I was writing my post. What you stated here is the consequence, and the cause is my last bolded question above, I think. Or maybe not... ;D

I have idea how to prevent this, but first we need to reach an answer for 0=NULL
Title: Re: Using NULL in Advanced Search
Post by: rick.ca on March 06, 2010, 12:58:14 am
Quote
I have idea how to prevent this, but first we need to reach an answer for 0=NULL

I'm not sure, but since the whole NULL thing is apparently up to Firebird and outside of nostra's control, maybe the answer is to initialize all numeric fields with "0" so there are no NULL's. And maybe for string fields, the Firebird NULL should be replaced (in effect) with a programmed NULL that means "empty" (perhaps determined by a string length function returning 0?). I have no idea what the performance implications might be, even if something like this were feasible. We might prefer the illness to the cure. :-\
Title: Re: Using NULL in Advanced Search
Post by: buah on March 06, 2010, 02:03:17 am
Exactly!
Because this is what scares me regarding PVD
Quote
since it allows a column to have a NULL value, it is definetely not a relational table
And, the most painless thing is if we're wrong here about NULL. But, what if we're not?

Quote
I have no idea what the performance implications might be

1. Since it is obvious that it's not possible to import NULL condition to PVD, but only empties, performance implication while importing data to PVD IS NULL. The real and genuine IS NULL one ;D

2 While generating new record within PVD, performance implication would be measurable in seconds, since generating full record takes several tens of seconds, right? And PVD still would be the fastest thing I ever used.

What do you think?

I mean, I'm running my 842MB PVD database on my 8 years old PIV 1,5GHz desktop without any performance problem. It's still unbelievable to me. That's why I'm inter alia so amazed with it.

Title: Re: Using NULL in Advanced Search
Post by: mgpw4me@yahoo.com on March 06, 2010, 02:26:21 am
If you look at the situation in binary, there is no numeric value for NULL, so numeric fields effectively can't be NULL unless there is a 'special bit' defined that says the number is NULL, and that would be proprietary / non-portable across applications.

In ASCII, the number '0' is defined as NULL. ( ASCII table reference: http://www.asciitable.com/ ) Strings (usually) end with a null (zero-terminated).  If the first byte of a string is 0, then the string is empty (has 'no value').

To get a true NULL value, go into the scripting language:

var somestring : string;  

This is a TRUE NULL value...it has not been assigned a value, but a place in memory has been reserved for it.  If you try to use 'somestring' before you set a value, the compiler will tell you that you're trying to use an uninitialized variable and your script will not compile.

In a database, remember that the values are loaded into memory, so a null value (depending on the database, and the field type) may result in the value 0 or it may simply leave the value already at that address in memory unchanged.  I have no idea how Firebird handles this.

For clarity, I'd prefer not to have null values...fields should be set to 'something' even if it is 'no value' (ascii code 0, which is '0' in a numeric field and "" in a text field).  In a date / time field, I don't know what validation checking is done, but I suspect the time is stored in a numeric format (unsigned long integer, based on an arbitrary starting date), so even setting the field to '0' will mean the date / time is the starting date, not a 'null' date...in which case NULL might be the only way to deal with it.
Title: Re: Using NULL in Advanced Search
Post by: AimHere on March 06, 2010, 02:54:34 am
Another follow-up to my earlier post.

I tried doing searches based on the "Budget" field. This field, like "Rating", is numeric... you can type any text you want into it in Edit Mode, but only numeric values will be saved when you save the record (non-numeric input will be "silently" rejected).

Now, it so happens that NONE of my movies have any budget data recorded for them in my database... "Budget IS NULL" returns 1364 out of 1364 records. But "Budget IS NOT NULL" returns 403 records!

In this case, the anomaly can be explained by this: When I first started using PVD, each new record added had a blank "Budget" field, whether I added the movie manually, by "New Movie Master", or by scanning files on disc. So those records have a truly NULL value for "Budget". BUT, at some point (possibly around the end of May '09), the behavior of PVD changed so that new records got a "Budget" value of ZERO if not otherwise specified. (A "Budget" of zero does not appear in the normal viewing mode, but will show up in Edit Mode, i.e. the field will actually have a "0" in it, as opposed to the older records where "Budget" is BLANK).

So, those 403 records are merely the ones that have true zeroes in "Budget", as opposed to the earlier NULL value. A search for "Budget = 0" returns the SAME 403 records.

The same thing goes for the "Box Office" and "Year" fields... earlier releases of PVD left them NULL if not specified, while newer versions set them to zero.

Note the difference between this behavior and that of the "Rating" field, as discussed earlier. For "Rating" and its siblings, "Rating = 0" and "Rating IS NULL" are equivalent (returning the same group of records), but that's not the case for "Budget"! "Budget = 0" returns ONLY the movies that have a budget actually equal to zero, NOT the ones that are NULL. "Budget IS NULL" returns movies with either zero or NULL budgets!

(Remember, I think that PVD always stores a numeric zero for unrated movies, even if that zero isn't displayed in Edit Mode, which is why "= 0" and "IS NULL" are equivalent for "Rating". "Rating IS NULL" is still returning records with either zero OR NULL values, but since "Rating" is never truly NULL, the condition simply returns all records with a rating of zero. That's not the case for "Budget", etc., which, as I've found, can have a truly NULL value, at least on older records.)

What it boils down to is this: the "IS NULL" condition will return records with (numeric) zero or (text) blank values AS WELL AS true "null" values, no matter what field you're searching on.

The "NOT NULL" condition always returns all records for which the tested field has ever been assigned ANY value (other than true NULL), regardless of the field's current value.

As mgpw4me has pointed out, there really is no way to set a NULL value on any field in PVD. Not-NULL values are created anytime a value is assigned to a field, and remain not-NULL forever, even if the field is cleared out or set to zero.

So if anything, the "IS NULL" testing is technically broken, as it technically should only return records for which the field is truly NULL (has never had any value assigned by either PVD or the user), not fields containing numeric zeros or empty strings. "IS NOT NULL", on the other hand, actually works as intended.  :P

Aimhere
Title: Re: Using NULL in Advanced Search
Post by: buah on March 06, 2010, 04:36:58 am
Quote
I think that PVD always stores a numeric zero for unrated movies, even if that zero isn't displayed in Edit Mode, which is why "= 0" and "IS NULL" are equivalent for "Rating"

For me, this is weak spot and I'll be free to try to clarify it further.
If you try to assign value "0" to "Rating" in edit mode you won't succeed, it'll silently become blank. Why? Isn't rating a number field, and isn't zero a number? The only logical reason for me why is that so is that it was meant for value "0" to mean "Unrated" and that "Unrated" is treated like NULL, and that's how "0" became "NULL".
How I see things, Value was equaled with Condition.

Orescb already noticed this here (http://www.videodb.info/forum_en/index.php?topic=1055.20) 13months ago and I found this after we conclude it here almost exactly the same as he was back then.
Nostra asked his suggestion, he responded, but no feedback, neither changes in PVD.

The fact that after 13 months a "new generation" of devoted users again recognized this as an possible issue should put this on "to revise list".

My suggestion for "Rating"
0. Default state is blank (NULL)
1. "0" is allowed (Value)
2. "Blank" is allowed (NULL)
3. Rating in View mode means that the one wants REALLY to rate the movie. So, any action in view mode can produce only values (0-10).
4. If one wants to "unrate" the movie, he could do it only in Edit mode by clearing rating - "blank"

Other numeric fields:
5. Default box office is blank (NULL) - We still don't know it
6. "0" is allowed (Value) - The producers committed suicide
7. "Blank" is allowed (NULL) - We don't want to know, or we aren't wanted to know how reach they became. (it isn't allowed at the moment!)

This way, all numeric fields would act identically. They would all have the same "bug" (ref. Aims testings). And I'm afraid this would be only the start point for the questions of all questions:

How's 0=NULL?*




*Or "Where's the pie?", or "Where's the money?" ;)
Title: Re: Using NULL in Advanced Search
Post by: rick.ca on March 06, 2010, 05:38:18 am
Wow. I'm really impressed with the thoughtful analysis of the problem. But now for the fun part. What should be done about it? I'm pretty sure it's not going to help much to make the program comply with the mathematical or database system meanings of NULL. It should behave as a normal non-technical user expects it to in the circumstances.

For text fields, NULL should mean "empty" (as in, "when I look at it, there's nothing there"). And perhaps it should be replaced with the the term EMPTY to avoid confusion. For numeric fields, NULL should have the same meaning as "0", even though in some contexts the two can be construed to mean something different. Like the question of whether a rating=0 should mean "unrated" or literally "0", I think we can do without the latter. I would be happy to see any NULL operators disappear and just use the standard arithmetic operators with "0". If it helps those who have a hard time using "0" to mean "empty" or "not set" (e.g., a rating or date), placebo EMPTY operators could be provided—they would give the same results as "= 0" and "<> 0".

I haven't thought this through very thoroughly. Am I missing anything? Are there any circumstances that would warrant exceptions to my suggested "rules"? But I have considered...

Quote
For me, this is weak spot and I'll be free to try to clarify it further.

I'm not ignoring you, buah, but you posted this after I drafted the above. I'll just add, I disagreed with Orescb 13 months ago, and I still do. I just don't think trying to accommodate distinctions between NULL, EMPTY and 0 are worth the trouble or the confusion they would cause many users even if implemented "perfectly."

I do realize, in general, there is one useful distinction between NULL and EMPTY. That's where NULL means I have no data—but there may be some to be had (e.g., the field hasn't yet been updated), while EMPTY means there is no data (e.g., the field has been updated, but there's no data available). The distinction is useful in the process of updating the database. But in any situation where this is important to me, I have the option to use other means to make the desired distinction—in more explicit terms. I will, for example, sometimes record an "n/a" (i.e., "not available" or "not applicable") in situations where I have looked for data and found none. In some numeric fields, I've used a "-1" to indicate the same thing. Even for something like My Rating, I have the option of deciding 0 means "unrated," 1 means "I've seen it, but I can't make up my mind," and 2 means "the worst movie I've seen in my entire life." If I want to be picky about the last one, instead of 2 I could use 1.4—the lowest rating on IMDb.
Title: Re: Using NULL in Advanced Search
Post by: mgpw4me@yahoo.com on March 06, 2010, 06:09:19 am
Personally, I don't find a difference between 'no data' and 'uninitialized'.  I still don't have data.  NULL just confuses the issue. A string comparison to "" would be better, in my opinion, than a comparison to NULL and I can live with the "huge" confusion caused by a box office of 0...sorta obvious that 0 is really NULL. 

Processing of dates is a different game.  Without NULL, you can't tell if the timestamp record is 0 or NULL.

Of course, we all realize that changing the database format means there will be a need for a conversion utility (to correct older databases) and since this project is not a democracy, we probably shouldn't volunteer Nostra's time until he has had his say.
Title: Re: Using NULL in Advanced Search
Post by: rick.ca on March 06, 2010, 07:39:53 am
Quote
Processing of dates is a different game.  Without NULL, you can't tell if the timestamp record is 0 or NULL.

Help. This sounds important, but I can't think why. Can you give an example of where this is important—in the UI?

Quote
Of course, we all realize that changing the database format...

Actually, I don't think it occurred to me what we're talking about implies a change in the database format. Maybe it does, but I wasn't thinking beyond the behaviour of the UI—which I assumed would be determined programmatically.

Quote
we probably shouldn't volunteer Nostra's time until he has had his say.

This always goes without saying. Our job is to enumerate all his choices before he has time to think about which TODO list any option might be added to. After this one, I won't be surprised if the infamous "Things to do when I have nothing better to do" list is replaced with a "IS NOT NULL" list.  ;D
Title: Re: Using NULL in Advanced Search
Post by: buah on March 06, 2010, 03:01:04 pm
I'm pretty sure that this topic won't help to get rid of NULL. And I'm sure that PVD will get more and more new users that already have some collections. Because of those users who'll import their collections to PVD in the future, I rather see this topic as a very useful to have it in mind while updating database, and please reconsider it to become sticky.

God knows how many records I updated using NULL and NOT NULL while importing my 5500 movies to PVD from other collections (I insist to repeat everywhere the number of my movies because of new users, to encourage them to do the same, cause it's easy and worthy).

I wish I had such a topic in that time. As such, it is more important to have it in Support, than in Feature forum, in my opinion.
Title: Re: Using NULL in Advanced Search
Post by: mgpw4me@yahoo.com on March 06, 2010, 03:49:54 pm
Help. This sounds important, but I can't think why. Can you give an example of where this is important—in the UI?

I'm thinking more of birthdays and dates of death than anything.  It isn't critical information since it's unlikely I'd sort by either field, but it makes a nice supplementary fact (like place of birth).  Since there is limited manipulation of these values, a string value would probably work...age determination would require conversion, but I can't think of anything else.
Title: Re: Using NULL in Advanced Search
Post by: AimHere on March 06, 2010, 08:26:15 pm
I'll let you guys try to sort this out.  ;D

If it were up to me, I'd like to see "NULL" mean a blank or empty field (including fields that at one time had data but which are CURRENTLY blank/empty), and "NOT NULL" mean anything else (including numeric "zero"). Further, clearing/blanking a field should reset it to NULL status, unlike now.

And the conditions "IS NULL" and "= 0" should NOT be equivalent. A value of zero is still a value, not a "null". Then "null/not null" would have NO relation to "zero" (or any other value) other than the obvious (i.e. if the field contains "zero", then it is NOT "null").

For this to be truly useful, though, PVD would have to leave ALL fields in a new record empty/blank/NULL unless otherwise specified, rather than filling in Year/Budget/BoxOffice with numeric zero like it does now. (Remember, at one time those three fields WERE left empty or "null" by default, unlike now.)

And the various "Rating" fields should make a distinction between "blank/null/unrated" and "zero", instead of treating them as equivalent! That way, a user could actually give a movie a rating of zero, without it being confused with an UNrated status. Perhaps the user interface could be modified to "ghost out" the "star bar" next to "Rating", or simply say "Unrated", for blank/null ratings. Any rating value (from 0 to 10) would cause the stars to appear normally (including all-empty stars for a zero rating), while clearing out the text box would reset the "star bar" to ghosted/unrated status.

This is the only behavior that would really make sense to me, and I suspect, the majority of users.

Aimhere
Title: Re: Using NULL in Advanced Search
Post by: rick.ca on March 06, 2010, 08:47:08 pm
Quote
God knows how many records I updated using NULL and NOT NULL while importing my 5500 movies to PVD from other collections

My experience may be different from yours for because...

In this case, the anomaly can be explained by this: When I first started using PVD, each new record added had a blank "Budget" field, whether I added the movie manually, by "New Movie Master", or by scanning files on disc. So those records have a truly NULL value for "Budget". BUT, at some point (possibly around the end of May '09), the behavior of PVD changed so that new records got a "Budget" value of ZERO if not otherwise specified. (A "Budget" of zero does not appear in the normal viewing mode, but will show up in Edit Mode, i.e. the field will actually have a "0" in it, as opposed to the older records where "Budget" is BLANK).

But whatever the reason, I've always found the technique of limited use if for not other reason than I have to question the results. I might find the idea of leaving NULL as it is more acceptable if we had more control over the relevant values. Maybe that will come with a new Multiple Movies Editor that allows any field to be set to NULL, blank (text) or 0 (numeric). That would allow me to use NULL to indicate "I see no cheese" and the others would mean "there is no cheese." It's still not very convenient, however, because NULL will still include the empty/0 values (I have to bookmark them to distinguish them). And then there's the issue of NOT NULL not returning a result intuitively consistent with the NULL result.

But it seems that not even this is feasible, because...

Quote
As mgpw4me has pointed out, there really is no way to set a NULL value on any field in PVD. Not-NULL values are created anytime a value is assigned to a field, and remain not-NULL forever, even if the field is cleared out or set to zero.
Title: Re: Using NULL in Advanced Search
Post by: buah on March 06, 2010, 09:16:45 pm
All what we said here made me thought of "NULL" as a "application/database is not aware of it" and for NOT NULL as a "application/database is/became aware of it", rather than unknown/known, empty/not-empty, blank/not-blank, etc...
Title: Re: Using NULL in Advanced Search
Post by: rick.ca on March 06, 2010, 09:42:30 pm
This is the only behavior that would really make sense to me, and I suspect, the majority of users.

You've got to be kidding! The majority of users aren't even aware of the issue or haven't found a reason to care. Most of the rest are confused because the technical operation of the NULL operator does not yield a result that is intuitive in the circumstances. It's the lack of consideration of what is sensible to the average non-technical user that is the source of the problem in the first place.

Quote
And the conditions "IS NULL" and "= 0" should NOT be equivalent. A value  of zero is still a value, not a "null". Then "null/not null" would have NO relation to "zero" (or any other value) other than the obvious (i.e. if the field contains "zero", then it is NOT "null").

This is technically correct, but doesn't have any bearing on the question at hand, until it's decided it should. I contend that it should not. As I've pointed out a number of times in a number of different ways, there's no reason why NULL and 0 should not be equivalent—if that is simpler and appropriate for the situation at hand. I'm not suggesting there's no circumstance in which different meanings would be appropriate—but they do seem difficult to find. For most users in most situations, they both mean the same thing, and any attempt to make a distinction therefore only causes unnecessary confusion.

Quote
And the various "Rating" fields should make a distinction between "blank/null/unrated" and "zero", instead of treating them as equivalent!

And this is the perfect illustration of my general point. There's absolutely no justification for the assertion there "should" be "zero" rating. As a rating scale, there's nothing wrong with it starting at 1. It is, in fact, far more sensible than what you suggest. Using stars alone, there's no practical way to distinguish between NULL and 0. And there's no reason to rate something 0 when you can just as easily decide 1 has the exact same meaning. It's also obvious most users will find it perfectly intuitive the way it is. With a 0 rating allowed, we'd be forever explaining the distinction between "0" and "unrated."
Title: Re: Using NULL in Advanced Search
Post by: rick.ca on March 06, 2010, 09:55:55 pm
All what we said here made me thought of "NULL" as a "application/database is not aware of it" and for NOT NULL as a "application/database is/became aware of it", rather than unknown/known, empty/not-empty, blank/not-blank, etc...

That's not surprising. But it seems too technical a nuance to be useful to most users, whereas the latter are well understood and expected. If it can coexist with functions that most users need and find intuitive without creating unnecessary confusion, then fine. Otherwise, I think we should be looking for ways to simply things and make the search functions more user-friendly. Any functionality that may be lost will likely only be noticed by those of us geeky enough that we're able to find another way to do what we want to do anyway.
Title: Re: Using NULL in Advanced Search
Post by: mgpw4me@yahoo.com on March 06, 2010, 10:23:36 pm
Clearing a movie sets all the values to default, so it IS POSSIBLE to set a NULL value in PVD (if the default value is NULL).  The description field is an example.

Regardless, I either have information or I don't.  I don't care if a script doesn't return a value...it's the same thing...there isn't any data.  To me, NULL / NOT NULL are not particularly useful and they confuse the issue...do you have data or not?  

The only place I can find where NULL has value is in setting a birthday or date of death, and those can both be text fields with a zero length...conversion to numeric values could be done easily enough if you really need to know the age of a person and can't do the math yourself.  Given that dates are text values when the script gets them, and are converted to a date value when submitted to the database, I see this as a better solution that having nulls for a single instance.

A rating of 0 is the same...it can't be set via IMDB...it isn't allowed by their database.  0 = no rating.  I have more than my share of movies where users commented they would have rated it lower, if it were possible.

On the programming end, I can see that nulls do have value.  It's easier to update a field if there is a standard value (null) for all field types (in particular when you have the option to have a field updated always, if no data exists, or never).  The PVD interface is already more than complex enough and should be simplified where possible.
Title: Re: Using NULL in Advanced Search
Post by: rick.ca on March 06, 2010, 11:10:25 pm
Quote
I have more than my share of movies where users commented they would have rated it lower, if it were possible.

I didn't say it in my last post, but now I can't resist... It's not hard to see the virtues in a -10 to 10 scale, where negative means degrees of "bad," positive means degrees of "good" and "0" means indifferent. But, aside from being just plain silly, that's not the scale that has been chosen. I suppose these people might also comment, "Roger Ebert gave this 1/2 star—he must have liked it a little." ;D
Title: Re: Using NULL in Advanced Search
Post by: mgpw4me@yahoo.com on March 06, 2010, 11:34:05 pm
You don't know the half of it...here's what I'm up against  ;)

[attachment deleted by admin]
Title: Re: Using NULL in Advanced Search
Post by: buah on March 06, 2010, 11:48:07 pm
All this fuss started because I wanted to RATE my seen UNRATED movies.
This was my thought process.
"How to get unrated movies? Their fields are blank, so filter NULL seen movies. OK, here they are. Nice... So, let's check if rated movies are "seen minus NULL". I'll apply NOT NULL, that far out makes sense!"

And that is when curiosity killed the cat.

If PVD was designed to start rating form 1, I assure you I'd apply Asearch "Rating<1" for getting unrated movies. I would never think of NULL, and I'd still live in ignorance, blessed it was!


[EDIT] mgp, you posted while I was typing my post. Now, I look at your IMDb statistics (man, you have way too many bad movies ;D) and new question arises. In legend, for example "4-6", "6-8" for IMDb rating and especially for personal rating in what count rating 6.0 is included?
Title: Re: Using NULL in Advanced Search
Post by: mgpw4me@yahoo.com on March 07, 2010, 12:01:35 am
In life, one sometimes steps in a cow pie then wonders why the party hosts refuse them entry.  It's a strange thing this "life".
Title: Re: Using NULL in Advanced Search
Post by: buah on March 07, 2010, 12:12:15 am
Indeed strange. And shorter than we're ready to admit. But, at least IS NOT NULL...
Title: Re: Using NULL in Advanced Search
Post by: mgpw4me@yahoo.com on March 07, 2010, 12:23:21 am
Not null = not unknown...hmmmm...
Title: Re: Using NULL in Advanced Search
Post by: buah on March 07, 2010, 12:30:33 am
See how my comprehension of NULL is more suitable?

Life IS NOT NULL = not (not aware of it)... ;)
Title: Re: Using NULL in Advanced Search
Post by: rick.ca on March 07, 2010, 12:36:29 am
You don't know the half of it...here's what I'm up against  ;)

Wow. No wonder you're so ornery. You should watch better movies. ;D

There may be hope for you. Since I started using PVD the proportion of movies with an IMDb rating < 6 that I've viewed has dropped to about 3% from the 23% it was in the two years before. And most cases, I viewed low-rated movies very intentionally. For example, I decided to watch the entire Planet of the Apes series, knowing full well a few of the movies were very bad. I recently watched Plan 9 from Outer Space (http://www.imdb.com/title/tt0052077/) (frequently billed as "the worst movie of all time," but getting a generous 3.6 on IMDb) because it was the subject of 8.0-rated Ed Wood (http://www.imdb.com/title/tt0109707/).

Topic? What topic? Good grief, why is there never a moderator when you need one? :o
Title: Re: Using NULL in Advanced Search
Post by: buah on March 07, 2010, 12:44:52 am
Topic? What topic? Good grief, why is there never a moderator when you need one? :o

When there's no cat, mice rule the game ;D

Quote
If PVD was designed to start rating form 1, I assure you I'd apply Asearch "Rating<1" for getting unrated movies. I would never think of NULL, and I'd still live in ignorance, blessed it was!

Was it worthy to repeat it?
Title: Re: Using NULL in Advanced Search
Post by: mgpw4me@yahoo.com on March 07, 2010, 12:47:19 am
Topic?  We don't need no topic !!  We already have said everything that needs saying.
Title: Re: Using NULL in Advanced Search
Post by: mgpw4me@yahoo.com on March 07, 2010, 01:02:51 am
See how my comprehension of NULL is more suitable?

Life IS NOT NULL = not (not aware of it)... ;)

Your are too cool  8)
Title: Re: Using NULL in Advanced Search
Post by: buah on March 08, 2010, 07:53:42 am
Actually, that's what I heard about you, Canadians. Confirmed, so far. :)
Title: Re: Using NULL in Advanced Search
Post by: rick.ca on March 08, 2010, 09:09:55 am
Now that explains why he watches so many bad movies! For six months a year, he can make himself NULL by going outside in his underwear for 30 seconds. I, on the other hand, am torn between movies and strolling on the beach pretty much year round. 8)
Title: Re: Using NULL in Advanced Search
Post by: mgpw4me@yahoo.com on March 08, 2010, 02:45:15 pm
I may have to update my movie rating system.  Sex, Violence, Language = 3 star movie....
Title: Re: Using NULL in Advanced Search
Post by: AimHere on March 19, 2010, 02:58:50 pm
This is the only behavior that would really make sense to me, and I suspect, the majority of users.

You've got to be kidding! The majority of users aren't even aware of the issue or haven't found a reason to care. Most of the rest are confused because the technical operation of the NULL operator does not yield a result that is intuitive in the circumstances. It's the lack of consideration of what is sensible to the average non-technical user that is the source of the problem in the first place.

I suppose so. I'm only saying "if it were up to me"... ;D

Quote
Quote
And the conditions "IS NULL" and "= 0" should NOT be equivalent. A value  of zero is still a value, not a "null". Then "null/not null" would have NO relation to "zero" (or any other value) other than the obvious (i.e. if the field contains "zero", then it is NOT "null").

This is technically correct, but doesn't have any bearing on the question at hand, until it's decided it should. I contend that it should not. As I've pointed out a number of times in a number of different ways, there's no reason why NULL and 0 should not be equivalent—if that is simpler and appropriate for the situation at hand. I'm not suggesting there's no circumstance in which different meanings would be appropriate—but they do seem difficult to find. For most users in most situations, they both mean the same thing, and any attempt to make a distinction therefore only causes unnecessary confusion.

I see your point.

Quote
Quote
And the various "Rating" fields should make a distinction between "blank/null/unrated" and "zero", instead of treating them as equivalent!

And this is the perfect illustration of my general point. There's absolutely no justification for the assertion there "should" be "zero" rating. As a rating scale, there's nothing wrong with it starting at 1. It is, in fact, far more sensible than what you suggest. Using stars alone, there's no practical way to distinguish between NULL and 0. And there's no reason to rate something 0 when you can just as easily decide 1 has the exact same meaning. It's also obvious most users will find it perfectly intuitive the way it is. With a 0 rating allowed, we'd be forever explaining the distinction between "0" and "unrated."

Also true enough.

I'd settle for having "IS NULL" versus "IS NOT NULL" testing return consistent, intuitive results regardless of which field is being tested.

Aimhere
Title: Re: Using NULL in Advanced Search
Post by: CAD on March 22, 2010, 01:50:15 am
Quote
There's absolutely no justification for the assertion there "should" be "zero" rating. As a rating scale, there's nothing wrong with it starting at 1. It is, in fact, far more sensible than what you suggest. Using stars alone, there's no practical way to distinguish between NULL and 0. And there's no reason to rate something 0 when you can just as easily decide 1 has the exact same meaning. It's also obvious most users will find it perfectly intuitive the way it is. With a 0 rating allowed, we'd be forever explaining the distinction between "0" and "unrated."

Code: [Select]
Enter Devil Advocate ModeThere is nothing wrong with having a rating scale starting at 0.
If a movie is a real stinker 0 is perfectly valid.

null and 0 are different.
0 is a value - by definition - the value between + and - numbers
null is not populated. ie the field in question is empty. ( an invisible character eg "space" is a value and does not meet "null" criteria ).

I can see merit in having a rating system starting at 0 and null.
eg.
a movie is a stinker would be 0.
I haven't given this movie a rating yet would be null
Both should be independently searchable.

Quote
Using stars alone, there's no practical way to distinguish between NULL and 0.

perhaps if the movie is unrated " there is no spoon " err - there are no stars (displayed).

Code: [Select]
Exit Devil Advocate Mode
Title: Re: Using NULL in Advanced Search
Post by: rick.ca on March 22, 2010, 02:16:13 am
True to the Devil's Advocate tradition, you've presented a classic one-star argument that helps us all accept the foregone conclusion the rating scale is fine the way it is. 8)