Jump to content

"On the fly" feature option


Recommended Posts

Hello, 
After tinkering a bit with workflows, there is a feature i believe it is necessary:  "on the fly" option for tasks that involve any kind of transcoding.

At the moment, when trying to convert to a file a new location from "X" to "Y" format we are forced to choose between the following 2 ways:

Case 01) - read from storage 01 -> convert on storage 01 -> move to storage 02 

Case 02) - read from storage 01 -> copy from storage 01 to 02 -> convert on storage 02 -> delete copy on storage 02

 

copy_resources.thumb.png.b67af9ec470d2e1a97cdbd9a529d7fc6.png

 

 

 

 

 

 

It might seem pretty simple, but both scenarios create their own issues that could very easily solved with a "on the fly" option:

Case 01:
Seems pretty straightforward, but there is a problem:
The "original file to format Y " conversion creates traffic both ways (reading and writing) to the storage. And then a third time to move the conversion out of its original place. 

This is x3 amount of traffic on the network. 
In a 10gbE that might not be a problem, but on a WAN or even a gigaBit ethernet, that can be a show stopper.
Also, moving is a kind of "deleting" operation, that might not be "wise" on certain types of environments.

Case 02:
I believe this one will be the most usual, but still shows issues:

The original file gets copied to the second storage location, and then read again to be transcoded. 
Later, the copy of the original file gets deleted.

This creates, again, a x3 the amount of needed traffic on the network and temporarily x2 on the storage space. 


In both scenarios the process of conversion is going to work, but the stress the nertwork suffers is worthless.
In a few hundred megabytes file this is alright, no harm. But going to large sizes and large quantities of files, this becomes slow-to-almost-unusable.
Also, we may think that network & storage resources are always scarce.

 

The solution, in my opinion, would be to create a "on the fly" option, that uses the system internal memory, to handle the conversion without relying on the storage as such.

Broadly speaking, this would force the workflows to: read the original file -> store it, or a portion of it, on its memory for transconding -> transcode -> copy/render it to the destination location.

I believe that this should be an option for all the nodes involved in converting/transcoding/transforming any material. And that should be exposed on the face of the node, when the option is activated

 

Thanks a lot.

Miguel.

copy_resources.png

Link to comment
Share on other sites

Hi Miguel!

Thanks a lot for this feedback. I have proceed with this as with the other features.

However, I have some doubts regarding this feature. I mean, when transcoding from storage 01 to storage 02, you are not necessarily forced to make a copy first. You can directly send the transcode to be placed on a different storage. Have you tried that?

Cheers,

Cristóbal 

 

 

Link to comment
Share on other sites

Please sign in to comment

You will be able to leave a comment after signing in



Sign In Now
 Share

×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.