large KMZ asset decoding

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

large KMZ asset decoding

Jon Bradley-2
Running into some big script timeouts here and wanted to know what  
others have done to deal with it.

Loading a KMZ asset in is no problem and works fine. However, during  
the decode process, because PV3D is utilizing the nochump ZIP  
library, it gets stuck during the decode process quite often, even on  
ZIP archives of around 1 MB in size.

Has anyone modified the Inflater.as file to add a timer to decode  
large blocks instead of the whole thing at once (specifically the  
decode function)?

thanks,

jon

_______________________________________________
Papervision3D mailing list
[hidden email]
http://osflash.org/mailman/listinfo/papervision3d_osflash.org
Reply | Threaded
Open this post in threaded view
|

Re: large KMZ asset decoding

Matthew Giger
Jon,

I've never posted here before, but you've touched on something near  
and dear to my heart, KML. I've implemented just what you are asking  
for, but the work processing is tied in with my own game kernel so it  
would be too hard to extricate from my code to integrate it with your  
code.

Inflate.as was converted from the zlib source code at zlib-1.2.3/
contrib/puff/puff.c. My recommendation is to convert the C into  
Actionscript and modify it along the way so that you can choose to  
inflate each dynamic table based on how much time alloted is left.  
Then upon re-entry of the inflation code, just pick up where you left  
off. If you don't have a day to do that, you could try modifying  
Inflate.as itself to do something similar, but I noticed that there  
are some serious speed improvements that are included in puff.c.

I don't know what you are working on, but you might be interested in  
using EarthBrowser to do what you need. Basically it's Google Earth  
written in Flash. I'm set to release it shortly after I give a  
presentation about it at Google next week.

If you want a sneak peek at the website plugin take a look at: http://www.earthbrowser.com/media/ebtest/
The AIR app is available at: http://www.earthbrowser.com

Good luck with your project.

Matt Giger
[hidden email]



> Running into some big script timeouts here and wanted to know what
> others have done to deal with it.
>
> Loading a KMZ asset in is no problem and works fine. However, during
> the decode process, because PV3D is utilizing the nochump ZIP
> library, it gets stuck during the decode process quite often, even on
> ZIP archives of around 1 MB in size.
>
> Has anyone modified the Inflater.as file to add a timer to decode
> large blocks instead of the whole thing at once (specifically the
> decode function)?



_______________________________________________
Papervision3D mailing list
[hidden email]
http://osflash.org/mailman/listinfo/papervision3d_osflash.org
Reply | Threaded
Open this post in threaded view
|

Re: large KMZ asset decoding

Ralph Hauwert
That's incredibly good and well done. Applause!

ralph.

On 9 jul 2008, at 23:23, Matt Giger wrote:

> need


_______________________________________________
Papervision3D mailing list
[hidden email]
http://osflash.org/mailman/listinfo/papervision3d_osflash.org
Reply | Threaded
Open this post in threaded view
|

Re: large KMZ asset decoding

Ralph Hauwert
In reply to this post by Matthew Giger
btw, it is a mod of papervision3D that is running on ?

Regards,
Ralph.

On 9 jul 2008, at 23:23, Matt Giger wrote:

> Jon,
>
> I've never posted here before, but you've touched on something near
> and dear to my heart, KML. I've implemented just what you are asking
> for, but the work processing is tied in with my own game kernel so it
> would be too hard to extricate from my code to integrate it with your
> code.
>
> Inflate.as was converted from the zlib source code at zlib-1.2.3/
> contrib/puff/puff.c. My recommendation is to convert the C into
> Actionscript and modify it along the way so that you can choose to
> inflate each dynamic table based on how much time alloted is left.
> Then upon re-entry of the inflation code, just pick up where you left
> off. If you don't have a day to do that, you could try modifying
> Inflate.as itself to do something similar, but I noticed that there
> are some serious speed improvements that are included in puff.c.
>
> I don't know what you are working on, but you might be interested in
> using EarthBrowser to do what you need. Basically it's Google Earth
> written in Flash. I'm set to release it shortly after I give a
> presentation about it at Google next week.
>
> If you want a sneak peek at the website plugin take a look at: http://www.earthbrowser.com/media/ebtest/
> The AIR app is available at: http://www.earthbrowser.com
>
> Good luck with your project.
>
> Matt Giger
> [hidden email]
>
>
>
>> Running into some big script timeouts here and wanted to know what
>> others have done to deal with it.
>>
>> Loading a KMZ asset in is no problem and works fine. However, during
>> the decode process, because PV3D is utilizing the nochump ZIP
>> library, it gets stuck during the decode process quite often, even on
>> ZIP archives of around 1 MB in size.
>>
>> Has anyone modified the Inflater.as file to add a timer to decode
>> large blocks instead of the whole thing at once (specifically the
>> decode function)?
>
>
>
> _______________________________________________
> Papervision3D mailing list
> [hidden email]
> http://osflash.org/mailman/listinfo/papervision3d_osflash.org


_______________________________________________
Papervision3D mailing list
[hidden email]
http://osflash.org/mailman/listinfo/papervision3d_osflash.org
Reply | Threaded
Open this post in threaded view
|

Re: large KMZ asset decoding

Ralph Hauwert
In reply to this post by Jon Bradley-2
Sorry for hijacking your thread like this Jon. I'm pretty sure that a  
simple hack within the inflate loop of the zip libraries should do the  
trick....I had a quick look. It's all about saving a state when your  
timer hits, breaking the process, then coming back to it when needed.  
I wish I had time to help you out with it, since it's an interesting  
issue on itself, but currently I don't have enough to build a proper  
solution for it. Maybe dave (nochump zip library) can help out ?

Regards,
Ralph.
On 9 jul 2008, at 15:55, Jon Bradley wrote:

> Running into some big script timeouts here and wanted to know what
> others have done to deal with it.
>
> Loading a KMZ asset in is no problem and works fine. However, during
> the decode process, because PV3D is utilizing the nochump ZIP
> library, it gets stuck during the decode process quite often, even on
> ZIP archives of around 1 MB in size.
>
> Has anyone modified the Inflater.as file to add a timer to decode
> large blocks instead of the whole thing at once (specifically the
> decode function)?
>
> thanks,
>
> jon
>
> _______________________________________________
> Papervision3D mailing list
> [hidden email]
> http://osflash.org/mailman/listinfo/papervision3d_osflash.org


_______________________________________________
Papervision3D mailing list
[hidden email]
http://osflash.org/mailman/listinfo/papervision3d_osflash.org
Reply | Threaded
Open this post in threaded view
|

Re: large KMZ asset decoding

Roy Wiggins
While you're at it, I've had timeouts when loading collada files. Spreading it out over multiple frames would be really useful.

On Wed, Jul 9, 2008 at 5:57 PM, Ralph Hauwert <[hidden email]> wrote:
Sorry for hijacking your thread like this Jon. I'm pretty sure that a
simple hack within the inflate loop of the zip libraries should do the
trick....I had a quick look. It's all about saving a state when your
timer hits, breaking the process, then coming back to it when needed.
I wish I had time to help you out with it, since it's an interesting
issue on itself, but currently I don't have enough to build a proper
solution for it. Maybe dave (nochump zip library) can help out ?

Regards,
Ralph.
On 9 jul 2008, at 15:55, Jon Bradley wrote:

> Running into some big script timeouts here and wanted to know what
> others have done to deal with it.
>
> Loading a KMZ asset in is no problem and works fine. However, during
> the decode process, because PV3D is utilizing the nochump ZIP
> library, it gets stuck during the decode process quite often, even on
> ZIP archives of around 1 MB in size.
>
> Has anyone modified the Inflater.as file to add a timer to decode
> large blocks instead of the whole thing at once (specifically the
> decode function)?
>
> thanks,
>
> jon
>
> _______________________________________________
> Papervision3D mailing list
> [hidden email]
> http://osflash.org/mailman/listinfo/papervision3d_osflash.org


_______________________________________________
Papervision3D mailing list
[hidden email]
http://osflash.org/mailman/listinfo/papervision3d_osflash.org


_______________________________________________
Papervision3D mailing list
[hidden email]
http://osflash.org/mailman/listinfo/papervision3d_osflash.org
Reply | Threaded
Open this post in threaded view
|

Re: large KMZ asset decoding

Jon Bradley-2
In reply to this post by Matthew Giger
Matt,

> I don't know what you are working on, but you might be interested in
> using EarthBrowser to do what you need. Basically it's Google Earth
> written in Flash. I'm set to release it shortly after I give a
> presentation about it at Google next week.

Dude, that's proper. Nice touch with the satellites ... :) Kudos.

I'll definitely check out the puff.c code and see if I can port it.  
It would probably be better to do that than modify the nochump zip  
stuff myself. Man, I wish defate/inflate on ByteArray were included  
in the Player and not just AIR (and while I'm at it native decode/
encode of JPG and PNG ... ugh).

> Good luck with your project.

And yours as well!

I'm actually just doing some testing work to see how feasible  
something is right now.

I've got a whole separate project that I'm working on that retrofits  
a rendering engine (raytracing/radiosity) into PV3D. Working out the  
ZIP issues are going to be very useful for that from a data IO  
standpoint.

thanks for the tips!

jon


_______________________________________________
Papervision3D mailing list
[hidden email]
http://osflash.org/mailman/listinfo/papervision3d_osflash.org
Reply | Threaded
Open this post in threaded view
|

Re: large KMZ asset decoding

Jon Bradley-2
In reply to this post by Ralph Hauwert

On Jul 9, 2008, at 5:57 PM, Ralph Hauwert wrote:

> Sorry for hijacking your thread like this Jon. I'm pretty sure that a
> simple hack within the inflate loop of the zip libraries should do the
> trick....I had a quick look. It's all about saving a state when your
> timer hits, breaking the process, then coming back to it when needed.

No worries... thanks for hopping in though and pointing out that  
Matt's project rocked... he def. should get some attention on that  
piece.

I've already done similar things to the coreilb JPG encoder and PNG  
encoders, so I think doing it to the Inflate code won't be too much  
of a concern. It looks like only two funcs need to be messed with  
(and a couple vars with a timers) so I can't imagine it would take  
that long. :)

> I wish I had time to help you out with it, since it's an interesting
> issue on itself, but currently I don't have enough to build a proper
> solution for it. Maybe dave (nochump zip library) can help out ?

Heh... thanks for at least looking though. Generally, I prefer not to  
bug people to mod their code. It's probably best that I do it myself  
and contribute the mod back. :)

cheers,

jon

_______________________________________________
Papervision3D mailing list
[hidden email]
http://osflash.org/mailman/listinfo/papervision3d_osflash.org
Reply | Threaded
Open this post in threaded view
|

Re: large KMZ asset decoding

Jon Bradley-2
In reply to this post by Roy Wiggins

On Jul 9, 2008, at 9:32 PM, Roy Wiggins wrote:

While you're at it, I've had timeouts when loading collada files. Spreading it out over multiple frames would be really useful.

For the actual parsing of the Collada files, I don't know if it's fully implemented but it will asynchronously load the file and parse if it you tell it to.

yourDaeReaderInstance = new DaeReader(true);

You'll may need to do either do one of the following (I don't know when/where I got this info):

1. Modify org.ascollada.io.DaeReader to always use async (in the constructor).

2. Change the loadDocument method in DaeReader as follows:

public function loadDocument( data:* ):DaeDocument
{
this.document = new DaeDocument( data, this.async );
_numAnimations = this.document.numQueuedAnimations;
_numGeometries = this.document.numQueuedGeometries;
if( _numGeometries > 0 )
{
readGeometries();
}
else
{
dispatchEvent( new Event(Event.COMPLETE) );
}
return this.document;
}

3. If you prefer to keep the async option as false by default, you'll need to modify the Collada parsers themselves (ie, org.papervision3d.objects.parsers.DAE) to make sure the DaeReader instance (parser) is created with async, in the load method.

this.parser = new DaeReader(true);


There are a couple options with this.

cheers,

jon


_______________________________________________
Papervision3D mailing list
[hidden email]
http://osflash.org/mailman/listinfo/papervision3d_osflash.org
Reply | Threaded
Open this post in threaded view
|

Re: large KMZ asset decoding

Jon Bradley-2
In reply to this post by Ralph Hauwert

On Jul 9, 2008, at 5:57 PM, Ralph Hauwert wrote:

I'm pretty sure that a  

simple hack within the inflate loop of the zip libraries should do the  

trick....I had a quick look. It's all about saving a state when your  

timer hits, breaking the process, then coming back to it when needed.  

I wish I had time to help you out with it, since it's an interesting  

issue on itself, but currently I don't have enough to build a proper  

solution for it. Maybe dave (nochump zip library) can help out ?


Little update. I fixed up the classes yesterday and manage to get it to load in chunks quite well. ZipFile and Inflate now extend EventDispatcher and I've got a couple new event classes to handle when a entry is parsed (passed as a property of the event) and an error is thrown in the Inflate (passes the error). Also implemented ProgressEvent dispatch.

Still looking to see if I can simplify the code a bit more or streamline it.

Unpacking large ZIP archives is fine now. The next step is to modify the KMZ parser to queue the decompression of the entries. As long as the developer listens to the loaded event, all should work as it did before.

- jb

_______________________________________________
Papervision3D mailing list
[hidden email]
http://osflash.org/mailman/listinfo/papervision3d_osflash.org
Reply | Threaded
Open this post in threaded view
|

Re: large KMZ asset decoding

Drae
Jon Bradley-2 wrote
Little update. I fixed up the classes yesterday and manage to get it  
to load in chunks quite well. ZipFile and Inflate now extend  
EventDispatcher and I've got a couple new event classes to handle  
when a entry is parsed (passed as a property of the event) and an  
error is thrown in the Inflate (passes the error). Also implemented  
ProgressEvent dispatch.

Still looking to see if I can simplify the code a bit more or  
streamline it.

Unpacking large ZIP archives is fine now. The next step is to modify  
the KMZ parser to queue the decompression of the entries. As long as  
the developer listens to the loaded event, all should work as it did  
before.
This was a LONG time ago, but I just ran into this issue myself, and I am hoping that you are still around and willing share your code!

Thanks,
Drae