[opendtv] Re: A full explanation of the PSIP time issue.
- From: "Paul Freeman" <paul@xxxxxxxxxxxxx>
- To: <opendtv@xxxxxxxxxxxxx>
- Date: Thu, 24 May 2007 22:10:16 -0500
Hi John,
Thanks. I realize that there is still some disconnect between yours and my
understanding of this issue, but I haven't yet figured out what the cause of
this disconnect is, and thus which side is interpreting the spec correctly
and which one is not. I do realize that you have had discussions with some
leading members of the ATSC committees on this and you have actually
implemented the spec - so obviously you have given this a lot of thought. I
will continue to think about this to see if I can figure out why both sides
seem to think that their's is the correct way to interpret the spec but yet
disagree on the interpretation.
BitRouter does not have any transport streams from KYES-DT in our
collection. This is a good sign, as we typically only receive streams from
our customers when there are problems with the streams. Our customers then
want us to diagnose the cause of the problem and provide a work-around. No
one has reported any problems from KYES-DT to us.
In the interest of time I think I'll have to bow out of further discussion
on this debate now unless something suddenly clicks for me that tells me
that my interpretation is wrong. If so, I'll post an update then.
Best Regards,
-Paul
_____
From: opendtv-bounce@xxxxxxxxxxxxx [mailto:opendtv-bounce@xxxxxxxxxxxxx] On
Behalf Of johnwillkie
Sent: Thursday, May 24, 2007 8:59 PM
To: opendtv@xxxxxxxxxxxxx
Subject: [opendtv] Re: A full explanation of the PSIP time issue.
Hi Paul;
I don't think you've been argumentative, or at least less so than I.
Frankly, I don't see how this discussion impacts anything on the receiver
side. Also, I've never heard of anyone finding nits in the Sarnoff test
streams, but IIRC, they come from the time before A/65. I guess they've
been updated.
Do the streams tell you what the displayed time value is for a particular
system_time value? This might be helpful to this discussion, just as the
practical STT values entered into the spec on an informative basis in Annex
D a few revisions back are helpful. At least, they're helpful for my
argument.
If you take the system_time value, add it to UTC January 6, 1980 00:00:00,
then add in GPS_UTC_offset, you end up with GPS time (y,m,d,h,m,s). If you
take the system_time value and add it to UTC January 6, 1980 00:00:00, you
end up with UTC time (y,m,d,h,m,s). If this is what your systems do, then
it's entirely consistent with my understanding.
The issue is how you create the system_time value in the STT. If you start
your calculation with the GPS time value (which I think all agree is
advanced a set number of seconds from UTC), when you subtract January 6,
1970 00:00:00, you end up with a value that is greater than the system_time
value created by doing the calculation starting with the UTC time at the
same moment. And, the delta is equal - assuming everything is set correctly
- to the GPS_UTC_offset value.
My point - and I'm the only person who posts to this list who has EVER done
this - is that all time input into the STT is referenced to UTC and not GPS.
To associate it with GPS, the generator has to be provided in real-time with
the GPS_UTC_value along with the time values.. Nobody has addressed in any
of this simple fact: the display of GPS time for any time is ahead of the
UTC representation of the same moment, and this HAS TO BE TAKEN INTO ACCOUNT
IF YOU ARE CREATING SYSTEM TIME TABLES FROM GPS SOURCES. It doesn't have to
be taken into account if you are using a UTC source, since the time base is
UTC. You can create STTs from UTC time values directly, and you can create
them from GPS if you know the GPS_UTC_offset value that applies to that GPS
time value.
People who have never created an STT by hand or by code are telling me that
I misremembered something that I quoted on this subject, or that I
misunderstand the spec, or that their 'drive-by' perusal of the spec leads
them to believe I am wrong. Mark Eyer's example ONLY supports my position,
but nobody has the cohones to contact him or Art Allison or Graham Jones to
see if it's me that is wrong, or if there is a variety of opinion on this,
or if I am right. All of my attempts (all were approached neutrally, not
"am I right?") to do this with people more knowledgeable than I, or who were
there at the birth, or both, have ended up with my view being reinforced.
Hell, in small group discussions, Art Allison has even started off a
sentence with "you create STTs from the GPS time values." I challenged him
that all STTs had to be created using UTC values. He thought about it for a
second and agreed that I was right, that GPS values were used only to mark
the passage of UTC tme.
Now, this was several years before I called him up to ask him the same
question, and I don't fault him for not remembering: nobody can remember
all aspects of the spec at the same time, or at least can do it for very
long. And, there can be tension between "theoretical" views informed in the
creation of the spec, and "practical" views that arise when you try to
implement. Art tends to chuckle when I start down that path, but he can
keep me honest. I see no practical/theoretical tension on this.
It may seem like I have ego involved in this, but I really don't. If I am
wrong, I have one customer whose installation needs to be corrected, and
after about 60 minutes of coding and testing the code, I end up "being
right." And, the 60 minutes includes uploading the correction on my
customers's machine and testing it there. I've had some work of mine
subjected to review and peer review. I appreciate direct, pointed
corrections, even from sources I can't identify. (Many were from list
members who also criticized my usage in OpenDTV postings; that was
off-color.)
Otherwise, the GPS_UTC_value doesn't need to be provided dynamically, since
it only increments (at least so far) and one is given notice well in advance
of the change date/time for the new value.
Fun code? That's taking in the new GPS_UTC_offset value ahead of time,
store the new value, and change everything within the second that the change
applies.
I have the highest admiration for people who have instantiated the spec in
code. And, I hear you about dealing on the receive side with non-compliant
implementations. "Exception Handling" is very complicated.
Like you, I am always interested in taking in new information with an eye to
how it might affect my code.
And, if you have any transport streams from KYES-DT anchorage that have
PSIP, you have data output from my application. The generator is down there
right now having nothing to do with PSIP, but we should have it back up
within a week or two, after a reconfiguration of the transport chain is
completed.
John
_____
De: opendtv-bounce@xxxxxxxxxxxxx [mailto:opendtv-bounce@xxxxxxxxxxxxx] En
nombre de Paul Freeman
Enviado el: Thursday, May 24, 2007 5:43 PM
Para: opendtv@xxxxxxxxxxxxx
Asunto: [opendtv] Re: A full explanation of the PSIP time issue.
Hi John,
Thank you. I work for a company called BitRouter. We license a full ATSC
middleware stack to both silicon companies and to TV OEMs. Often a silicon
company needs to be able to supply ATSC middleware (PSIP, closed captioning,
and CableCARD SW) to their TV OEM customers in order to be able to get a
design win, and so they license our middleware and port it to their silicon.
I am BitRouter's VP of engineering as well as the author of our PSIP
software. Before that I wrote PSIP middleware for a company called
SoftProse, and before that I wrote PSIP and DVB-SI middleware for a company
called Microware.
I have no doubt that you have found many errors with EtherGuide Ferret in
transport streams being broadcast in the real world. We see the same thing
all the time and have invested a lot of engineering effort designing
work-arounds in our ATSC middleware so that our customer's TV sets continue
to work with the many illegal PSIP and captioning cases in the real world. I
have a stack of probably 100 DVDs full of illegal transport streams that we
have collected via field testing with our customer base and have enhanced
our SW to work with.
I am not trying to be argumentative with you on this issue, but instead am
just trying to understand whether our company has somehow misinterpreted the
PSIP spec. If so, I want to make sure that our middleware is fixed so that
it parses and interprets the STT correctly, as we pride ourselves on having
the most correct and robust receiver-side PSIP implementation in the
industry. So far I have not discovered the hole in our logic, if there is
one. But maybe the light bulb will still turn on.
I wish I could give you a list of TV models that use our software, but this
is difficult. There are two primary reasons for this. The first is that with
most of our customers we are under NDA and are not allowed to state that we
have provided the ATSC software for their silicon or their TV models. The
other reason is that when we license our SW to silicon companies, we don't
always know who their end TV OEM customers are. We may get reports on the
volume of chips they have sold and which our software was packaged with, but
we don't always know to whom they made the sale. Also, when we do know this,
we don't always know the model numbers of the TVs that the chips go in to.
i.e., Some TV OEMs use MPEG system-on-chip silicon from different vendors in
different models, so although we may know that TV OEM "xyz" is buying chips
from a company that licenses our software, we don't know which models these
go in to.
We have verified all of our middleware with the Sarnoff PSIP compliance test
streams, and these streams do contain some tests of the STT. According to
these streams we are interpreting the GPS_UTC_offset field correctly. This
is not to say that both the Sarnoff streams and our middleware are correct -
perhaps both are wrong. But this is about as close as we can come to a
formal verification. What we see in the real world is that the STTs carried
by broadcasters are typically not reliable enough to be able to verify our
STT parsing against real world broadcasts. For example, if I check the local
broadcasters here, the STT jumps back and forth among them, varying by as
much as two to three minutes from the lowest system_time value to the
highest system_time value at any point in time.
Anyway - You seem so sure of your argument that I am not giving up on your
theory. I do still want to think about it some more to see if there is
something I am missing in my interpretation.
_____
From: opendtv-bounce@xxxxxxxxxxxxx [mailto:opendtv-bounce@xxxxxxxxxxxxx] On
Behalf Of johnwillkie
Sent: Thursday, May 24, 2007 7:01 PM
To: opendtv@xxxxxxxxxxxxx
Subject: [opendtv] Re: A full explanation of the PSIP time issue.
Paul;
I've not just written a PSIP generator (EtherGuide Emissary ATSC), I've also
written a transport stream/metadata validator called EtherGuide Ferret ATSC
that ALWAYS (at least, every time so far) finds errors in transport streams
by comparing PSI to PSIP and to elementary streams and against A/65,
13818-1, 13818-2, A/69, A78 and outputs PMCP derived from the data.
Extensive user interface, presentation of data in hex, binary, decimal, and
as "EPG" data. More elegant than TS Reader, but much, much more expensive.
It's one of the reasons I say that I have mathematical proof. I can do this
closed loop: source data (pmcp or converted into PSIP) converted into PSIP
-> received, demuxed, converted into PMCP. Validated against the source
date. Tested against other implementations. Tested (at least tx) against
HP packet analyzers. Also, aside from two or three low-level routines (and
CRC-32 calculation), different code, written in different years, for encode
and decode.
Sorry if it seemed I spoke down to you. Until someone gives me their
bonafides on such a subject, I have to assume minimal PSIP knowledge.
Give me a list of the sets that include your PSIP, and I might be able to
tell you a bit as to how well they conform to the spec. There are quite a
few of PSIP implementations out there on the receive side that are
substandard or worse. I spot errors in encoder implementations, and even
other t&m gear, from time to time.
So, you believe that the UTC time value is to display the time to viewers?
Kindy kludgy to start (always) with UTC time, convert it to GPS, transmit
it, then convert it back to UTC before use, is it not?
Thankfully, my implementations aren't written into silicon.
Come to think of it, I'll send you an invitation to my PSIP list. No PR,
but more than a few PSIP experts. And, a much lower SNR than this list. I
don't think there's been a posting in more than a month.
John Willkie
_____
De: opendtv-bounce@xxxxxxxxxxxxx [mailto:opendtv-bounce@xxxxxxxxxxxxx] En
nombre de Paul Freeman
Enviado el: Thursday, May 24, 2007 4:14 PM
Para: opendtv@xxxxxxxxxxxxx
Asunto: [opendtv] Re: A full explanation of the PSIP time issue.
Hi John,
I realize that local time zone is set in the receiver. Just as you have
spent a number of years implementing the transmission side of PSIP, I have
spent a number of years (seven) writing different implementations of the
receiver side of PSIP. PSIP software that I have written has been licensed
by a number of different silicon customers and TV OEMs and is used in
several million TV sets. This level of experience does not mean that I am
right - as we both seem to have significant experience with PSIP and yet
only one of us is right. But I just wanted to point this out to you.
But my point had nothing to do with where the adjustment for local time zone
and daylight savings time is made. My point was to answer your question of
why the PSIP spec would require the broadcaster to transmit the
GPS_UTC_offset field if the system_time and EIT event start time were both
in GPS seconds. And my argument for requiring this is so that the receiver
can compute the correct local time for display purposes, for example in a
channel banner.
-Paul
_____
From: opendtv-bounce@xxxxxxxxxxxxx [mailto:opendtv-bounce@xxxxxxxxxxxxx] On
Behalf Of johnwillkie
Sent: Thursday, May 24, 2007 5:51 PM
To: opendtv@xxxxxxxxxxxxx
Subject: [opendtv] Re: A full explanation of the PSIP time issue.
Paul;
I KNOW that all times expressed in STTs are UTC. Until I had worked with
the spec for a few years, I believed that the times in STTs were GPS. I was
able to mathematically prove that I was wrong and I confirmed my new
information with several of the authors. I've talked about this with
multiple vendors and they have the same knowledge as I.
I'm reminded that at one time I thought it was unlikely that the length of
the diagonal of a right triangle could be calculated accurately by getting
the square root of the sub of the square of the length of the two other
sides combined. But, I was able to prove it mathematically in eighth grade,
so I haven't had much need to check the math in the meantime to see if
something changed.
I guess you've missed the rather (to me) clear point: if the time in STTs
was GPS, there is NO USE of the GPS_UTC_offset value, because the terminal
would just be able to tell the time by comparing the start_time (32 bits)
with the system_time ()32 bits).
It would be required, if the system time was UTC, since you would only be
able to match the start time of an event by adding system_time plus
GPS_UTC_offset and compare the result with the EIT event start time.
Your comments below evidence an understanding, but you have made no argument
for ANY need of UTC time in the receiver, if the STT and EITs are denoted in
GPS. If UTC is transmitted in the STT and EITs are GPS, then the need is
clear; you add GPS_UTC_offset to the UTC integer to get the GPS integer.
I need to point something out: local time zone is set in the receiver, and
the same correction for the local time zone (minutes and seconds) is applied
to GPS or UTC to derive local time. How to handle this is left up to
implementers, but it's a value that doesn't change during the time a set is
turned on, unless the set happens to be in a moving vehicle that crosses a
time zone boundary while being turned on. Adding or subtracting static
values like this is quite simple; things get fun when you have to add or
subtract dynamic variables on the fly.
So, local time has nothing to do with this 'argument.' Daylight savings time
data is transmitted in the STT only within 30 days of a change, other than
to indicate that DST is being observed or not.
Here's something else that is commonly misread in the spec. It says to
derive UTC, you subtract GPS_UTC_offset from GPS time. It doesn't say that
you need to do this, only that when you want to do that, you follow the
procedure.
But, Bert says that PSIP is simple.
Everything one needs to know about PSIP is contained in A/65 and the
normative and informative references. That doesn't mean that one can figure
out everything on the first dozen reads. Heck, it takes more reads than
that to spot some of the subtle ambiguities. However, that doesn't apply to
this situation, as there is NO AMBIGUITY in the spec on this point, there
are just people who haven't read it enough (and played with the math and
code) to figure it out.
Let me put it another way. After 'grokking' A/65, I found that other specs
were very simple to understand, almost at a glance, because of all the
subtle issues that I've figured out with A/65. (One exception: CEA-608 is
still a bit funky to me.) But then, when I'm reading another spec for the
first time, I tend to focus on step #3 - how do I use it?
John Willkie
_____
De: opendtv-bounce@xxxxxxxxxxxxx [mailto:opendtv-bounce@xxxxxxxxxxxxx] En
nombre de Paul Freeman
Enviado el: Thursday, May 24, 2007 3:05 PM
Para: opendtv@xxxxxxxxxxxxx
Asunto: [opendtv] Re: A full explanation of the PSIP time issue.
Hi John,
I think the reason for sending the GPS_UTC_offset is so that the receiver
can correctly determine the time of day for purposes such as displaying a
channel banner and for labeling the columns on an EPG grid display with the
correct start times. (I believe) the STT times are expressed in GPS and (we
both believe) EIT start times are also expressed in GPS within the PSIP
data. But the user wants to see these in times displayed in their local
time. So the receiver must first convert the transmitted GPS times to UTC
time by subtracting the GPS_UTC_offset, and then adjust the UTC time for the
local time zone and local daylight savings time status. Without transmitting
the GPS_to_UTC offset, the receiver would be displaying the wrong values in
the channel banner and grid guide.
_____
From: opendtv-bounce@xxxxxxxxxxxxx [mailto:opendtv-bounce@xxxxxxxxxxxxx] On
Behalf Of johnwillkie
Sent: Thursday, May 24, 2007 4:55 PM
To: opendtv@xxxxxxxxxxxxx
Subject: [opendtv] Re: A full explanation of the PSIP time issue.
Yes, I meant to correct that. EIT times are GPS time, and I erroneously
stated otherwise in this thread.
However, that just begs the point.
PSIP provides time of day, EPG and time for EPG events.
Since the time in EITs are GPS, and - a false notion - the STT is GPS, just
why is the GPS_UTC_value transmitted in the system time table? And, IT'S
REQUIRED TO BE ACCURATE!
It isn't needed; TV sets and receivers have no need for UTC (in this
argument). And, I can find no other circumstance in PSIP where 8 bits of
unneeded data is required to be transmitted, let alone where unneeded data
is required to have a value other than '1' in each bit position.
Now, that GPS_UTC_offset value is required in the actual PSIP time scenario,
to correct the transmitted UTC to GPS.
Once again, all this becomes obvious once you do the coding of just a PSIP
encoder.
John Willkie
_____
De: opendtv-bounce@xxxxxxxxxxxxx [mailto:opendtv-bounce@xxxxxxxxxxxxx] En
nombre de Paul Freeman
Enviado el: Thursday, May 24, 2007 1:46 PM
Para: opendtv@xxxxxxxxxxxxx
Asunto: [opendtv] Re: A full explanation of the PSIP time issue.
I am having a hard time concluding from the A/65C spec how the system_time
field in the STT is anything other than GPS time. I do not see how the
conclusion has been reached that this field represents UTC time, and so far
have been unable to understand the arguments that have been presented in
favor of the syste_time field representing UTC time.
But I have seen some of the posts claim that the EIT information is UTC
time, and nobody has refuted that yet. However, from my reading of the spec,
even though the EITs change at 12:00, 3:00, 6:00, and 9:00 UTC time (not GPS
time), the actual start times listed in the EITs are themselves expressed in
GPS time. That is, the "start_time" field in the EIT is described on page 44
of A/65C as "A 32-bit unsigned integer quantity representing the start time
of this event as the number of GPS seconds since 00:00:00 UTC, January 6,
1980".
Other related posts:
