Thursday, July 2, 2009

Request for comments on file transfer

As some of you have already noticed, all the networking stuff is disabled in my binary kernel image releases. This is because the A320 is not a networked device and as such in principle it doesnot make much sense to waste precious memory. Note that module loading is also disabled for the same reason (static hardware, reduces kernel memory usage). I'll later explain why all this is relevant.

File transfer is not yet enabled in the released kernels. The reasons for this are:
  • The storage device gadget driver (which would be the "standard way") is not tested yet.
  • The storage device gadget driver accesses the storage as raw block devices, and thus requires exclusive access to them. In other words: the filesystem is mounted by the host PC. This is why you can see the ext2/ext3 partition from a linux PC despite the fact that the original firmware does not support it.
  • Having a serial console is a MUST for application developers. This is provided by the serial gadget driver, which is enabled now (and has a hideous bug that crashes the kernel when unplugging the USB cable).
  • You can't use more than one gadget device simultaneously in the 2.6.24.3 kernel. Though I think this is solved in recent kernels by implementing composite gadget devices, backporting it to the 2.6.24.3 kernel is a huge task, and even if that is achieved, the composite device we would need (serial + storage) is not implemented yet even in the latest kernel.
To summarize: we need the serial console and it is not usable together with the storage gadget function. The obvious way to solve this is backporting composite gadget device support from recent kernels and coding a new serial + storage device.

I can think of several ther alternative ways to get what we need:
  1. Using a serial file transfer protocol over the serial gadget. This is rather inconvenient and definitely not for the end user. I know next to nothing about OBEX, but I'm given to understand that it can work also over a serial line. Maybe its usage would be simpler that a classic protocols for the end user.
  2. Module loading/unloading of the serial and storage gadget devices. This requires enabling module loading (which enlarges the kernel), is cumbersome to implement, and probably requires fixing first the above mentioned unplug bug.
  3. Establishing a network connection, either by using the ethernet gadget device or by using SLIP over the serial gadget (the first would be easier). Once the network connection is established you can telnet to get console access and you can transfer files via FTP or SCP. This requires enabling the kernel and user space networking stuff, and would probably be a bit inconvenient for the end user, since it would require to configure the gadget network card.
My guess is that backporting + coding a new composite USB device is the way to go. It's a fairly large task and requires good knowledge of the kernel USB stuff (which I don't have). It would take time, but I guess this is acceptable since right now we have a working way to transfer files.

All your comments and suggestions are welcome.
Posted by at 1:26 AM

14 comments:

  1. I believe that your proposal is the best in the long term.
    ReplyDelete
  2. But then requiring exclusive access to storage? Why not do as PMPs do: If USB is connected while startup, boot a special environment (kernel) with only the storage access.
    ReplyDelete
  3. There are several file transfer protocols supported in minicom. I guess they were used for BBS-es. The getty on the A-320 side probably needs support for it too.
    ReplyDelete
  4. I agree with Bastian, but maybe in this way developers won't have an easy-to-use debugging tool while developing this special environment (and this is the point of this RFC!).

    Anyway i think that the network connection capabilities wouldn't be too bad, because it would allow some (limited) functionalities such as on line gaming, and why not online installers. And maybe in a far far future if dingux will be able to run from the internal flash, a wifi connection through a sdio card...

    but maybe I'm flying a bit too high...
    ReplyDelete
  5. Booboo, do you really need a serial connection to debug, or is an ethernet connection OK too?
    I tink that full network access with g_ether would be far more useful: this way we would be able to synchronise files, download podcats...

    You forgot a possible alternative: kexec
    I tested it and it works (at least with vlinux images). It could be used to immediately reboot into the correct mode.
    The only requirement is to support kexec in the kernel. We could also have a special kernel for video playing, for instance. With kexec, the 'switch' is really fast, and could be hidden from the user with the 'quiet' option.

    Were you able to get g_ether to work?
    ReplyDelete
  6. Please add ethernet support via ethernet gadget device. This would be invaluable for development. telnet, ftp, samba, rdate and others. Thanks.
    ReplyDelete
  7. I have a kernel for XXX25 LCDs with gadget ethernet support, I got the USB0 showing on the Dingoo, but in dmesg on my PC, I got:

    [33992.192040] usb 4-6: new high speed USB device using ehci_hcd and address 3
    [33992.324788] usb 4-6: config 1 has 0 interfaces, different from the descriptor's value: 1

    lsusb gives me (on the PC side):
    Bus 004 Device 003: ID 0525:a4a2 Netchip Technology, Inc. Linux-USB Ethernet/RNDIS Gadget

    And I got no USB0 ...

    The kernel is at:
    http://cyberic.free.fr/Dingoo/zImage_g_ether,
    if anyone wants to try...
    ReplyDelete
  8. sorry for my ingorance, but are we talking here about "ethernet over usb" (aka without some form of hardware network device)?
    I think from "user perspective" mounting the internal and SD card as a disk would make the most sence. I doubt the "standard" end user will use sftp or so to put rom's on the device.
    If both can't be loaded at same time and the serial debugging (wich needs a modded dingoo) is usefull for developpers then why not enable this in a "dev" build?
    And make a other "general" package that for example has only the ability to detect a usb cable and then mount them as a disk?
    ReplyDelete
  9. How about forward-porting dingoo support into more recent kernels? This is the way to go, and if you manage to get it integrated into mainline, you'll have much less pain in the future if any new feature (or bugfix) appears that you want in dingux.

    Of course that would require to port Ingenic's patchset, which may be a lot of work. But in the end, it will be a win.
    ReplyDelete
  10. @aissen

    You haven't assessed the insane amount of work and -most important- knowledge that is required to forward port the Ingenic patches to a more recent kernel. Seriously, it's huge, and I don't have the knowledge, and though I'll be glad to learn (I would master the kernel internals), I don't have that much time.

    Note also that Ingenic will keep on updating their patchset for the 2.6.24.3 kernel and we would be forced to keep up forward porting the changes.

    And in the end, just consider what are -in our case- the REAL benefits of a recent kernel. I bet it's just more polished drivers in certain cases, like the USB gadget stuff, and if that's all, it's just way simpler to backport those drivers.

    @cyberic

    Of course, one of the advantages of the g_ether solution is that you don't need the serial connection. Just login vía telnet or ssh and debug (you can use the GNU debugger vía ethernet !!!).

    I did not know much about kexec and so it didn't come to mind. LOOKS LIKE A NEAT SOLUTION, which would work EXECTLY like the original kernel does: when USB cable plugging is detected, shutdown everything and kexec a new super-minimal kernel with just gadget storage enabled. However, THIS WOULD NOT DO FOR THE DEVELOPERS, which need remote console access (be it via g_serial or g_ether).

    Wait... I got it: add a setting that will disable automatic enter in mass storage mode. Leave g_serial or g_ether during normal mode and allow the user to manually change to mass storage mode.

    I went as far as you with the g_ether driver. I tracked the changes of the USB gadget stuff from the 2.6.24.3 kernel to the latest one and there have been quite a lot of fixes and improvements. Looks like the gadget stuff was still in its early days back when the 2.6.24.3 kernel was released, and that the g_ether driver is just quite buggy and does not work (at least in combination with recent desktop kernels).

    If we go with the g_ether solution, I'll backport all the fixes one by one until I get it working.

    @Polossatik

    Sure. That's a drawback of the g_ether solution: the average joe won't want to transfer files vía FTP or SCP. We want, if possible, a no brainer solution for them.
    ReplyDelete
  11. @Bastian

    Selecting serial or storage gadget at startup would work and would be easier to implement that the kexec solution (which I think would be a bit more functional).

    @mth

    Sure, minicom supports xmodem, zmodem and such, and adding support in the A320 rootfs is dead easy. But the point is that it is very cumbersome and not definitely for the end user (it is much worse than then FTP/SCP solution).

    The end user wants mass storage. If not possible at all, I would settle with ethernet.

    @batman52

    You raised a very IMPORTANT point: online installers. That would just be great. I don't think online gaming is worth considering at the moment, mostly because you would need a PC acting as a gateway and if you have a PC, why not just use it to play online with a way bigger screen?.

    However, WIFI WOULD BE AWESOME, and I'm just surprised to learn that there actualy exist miniSDIO WiFi cards, and they're not expensive. This is what I'll do:

    1- Find out if SDIO is possible in the A320.

    2- if (previous == yes) { order a miniSD WiFi card (donations will probably be enough to buy it); }
    ReplyDelete
  12. surely you should fix the usb-serial driver, then use standard usb gadget drivers for mass storage and ethernet drivers. Module loading/unloading isn't so much of a pain... arm 710 can manage it well enough at 27MHz without considerable slow down. When a user plugs in a USB cable you could ask if they'd like to enable mass storage driver and then pivot-root to a small ramdisk image so the sd card becomes available for remote mounting- would take a few seconds to load the ram disk, but not too bad.

    Or either back-port the USB stuff for simultaneous gadget devices or forward port the ingenic pathes, and you don't have to do module loading/unloading, but still would have to pivot-root or use kexec solution.

    Havent looked at USB stuff since 2.6.12 but I imagine fixing serial driver and using pivot-root and modules or kexec and a few different kernel binaries would be the easier than backporting the changes to the USB stack.
    ReplyDelete
  13. @booboo

    kexec is almost the same as reboot with specific options, right? But if you think that doing the detection on startup would be easier, then the only piece to the puzzle missing is just doing a reboot on USB-insertion.

    Those who want a serial console can just disable the reboot on USB, and all are happy
    ReplyDelete
  14. @booboo
    Here:
    http://lwn.net/Articles/291839/

    you can see that of lot of work was done in kernel 2.6.27. They even speak about composite driver.

    Maybe it would be easier to backport these features that forward-port all the ingenic patches...

    On the other hand, maybe more people would help for the solution 2, event ingenic maybe?
    ReplyDelete

Subscribe to: Post Comments (Atom)