PVR-150 IR stops responding (fixed!)

Update
This patch, with some fixes, has gone into the 0.4.2 release of ivtv. Please use ivtv 0.4.2+ rather than following the instructions here. You can get this from the ivtv site.

History
This is related to my IR blaster driver, in case that’s what you are looking for. Do read on though, it might be informative! If you are just looking for patch instructions, dive right to the end to skip all this detail.

A while ago I looked around to see if the hauppauge windows driver for the remote control had any issues. I found quite a few people who claimed that it stopped responding, and that they had to reset the hauppauge software, etc and one or two pointers to a new (beta) driver on the hauppauge site. This was fairly intriguing as among the release notes it lists “i2c (IR blaster) fixes”. I managed to find time this weekend to take the driver apart, and see exactly what they did. Basically they rewrote the i2c support.

Some background, in case you aren’t familar: i2c is a low speed (varies, say ~100-400KHz) 2 wire serial bus for interconnecting various random chips. Because it’s so useful, it actually gets used all over the place to ‘glue’ various chips together on a board and then let the OS driver control them. For example, on your typical PVR-150 card you’ll find at least tuner, video encoder, IR chip and eeprom (configuration data). These are all i2c connected, and the driver talks i2c to them to program them (as well as other memory mapped registers, but let’s stick to the issue).

In some cases, the i2c protocol is implemented by special hardware on one of the chips (e.g. the bt878a chip has an i2c controller in it). This is the nice case, since the CPU doesn’t have to do much in the way of work. In the cheap case, you get to control the clock & data lines yourself from the OS (a kind of winmodem approach!). The hauppauge stuff takes the latter approach — indeed so does a bunch of other hardware. It’s so common that there is a module in the linux i2c subsystem specifically for doing this — it’s called i2c-algo-bit, and the approach is known as ‘bit-banging’. As a side issue, bit-banging takes up some constant fraction of CPU, because it has to use busy delays (it’s not possible to get microsecond accurate sleeps in the kernel, so we must run the CPU in a loop). Since these loops are constant _time_ you’ll find that whatever the CPU the IR polling (for example) takes a constant amount of time. For lirc_i2c, this works out at about 1%, and for my lirc_pvr150 module (that sends a ‘poll’ command to the receiver as well to see if it is borked or not), it’s about 2%. This is why I think the hardware approach is better, but whatever, we are stuck with it.

From the looks of the hauppauge driver changes it appears that the classical bit banging algorithm causes problems with the IR chip that the PVR-150 uses. I disassembled both the new and old drivers, and the older driver uses something very similar to the approach of i2c-algo-bit — it uses KeStallExecutionProcessor, or a busy loop, to implement delays. The newer driver dispenses with this, and instead polls the lines to see when they have become high/low and uses memory mapped register reads for stalling. I’m guessing that those reads are (roughly) constant time because they have to go across the PCI bus. I implemented the newer method, and low and behold this seems to solve my problems. I’ve not seen an IR chip reset by my lirc_pvr150 driver in 16 hours now (update: now 28), whereas I was getting them _much_ more frequently beforehand (dozens a day). This stops the remote dropping out if the chip has to reset, which it did do a few times a day for seconds at a time. So I’m happier. Anyway, if anyone else would like to try it, here’s the HOWTO part…

The HOWTO part

This patch is against the latest SVN version of ivtv. Things are changing rapidly with ivtv, and there is currently no really stable release (as you’ve probably realised). It might apply against the 0.3.8 release, if you try that and it fails let me know and I’ll post a patch against that version. It should go into the driver, eventually, if it really does sort out the IR chip issues.

Get the ivtv code from svn (you will need to apt-get install svn or whatever first!). See this page for detailed instructions.

svn co http://ivtvdriver.org/svn/ivtv/trunk ivtv

Download the i2c patch from the trac ticket for this issue (I will keep this up to date), or from from here. Apply the patch:


cd ivtv/driver
patch -p0 < i2c.patch
make && make install
cd ../utils
make && make install

Then reboot. Please _do not_ just rmmod ivtv & insmod ivtv — this won’t work. If you don’t want to reboot, rmmod _all_ of the ivtv modules first, especially cx25840. If you don’t do that, you’ll get an oops (then reboot, no biggie).
Note that this patch also covers section #3 of the IR blaster HOWTO, so you don’t need to try and apply both patches (if you do that, it will certainly fail to apply cleanly).

That’s it. You should stop seeing log messages related to resetting the IR chip, and the remote shouldn’t drop out at all. That’s the idea anyway. Any feedback would be appreciated!

HTH,

Mark

This entry was posted in Random. Bookmark the permalink.

20 Responses to PVR-150 IR stops responding (fixed!)

  1. scott says:

    Just did the new patch tonight, ill send you the results asap.

    Thanks again Mark!

  2. Scott says:

    Welp, ive been testing this since the 4th, and everything is working fine, no freezeups. everything is working fine! Thanks Mark – You Rock!

  3. J. Longman says:

    Have you tried the ivtv-0.4.0 release yet?

    I’m trying to apply the patch and the first one in ivtv-driver.c is failing – gpio.h is already included. Ah, if I don’t assume they are reversed and if I apply them anyways, the other hunk that fails is the gpio EXPORT_SYMBOL. Maybe your earlier patch was integrated?

    I’ll build and see what happens.

  4. mark says:

    Yes, the earlier patch to export the ivtv_reset_ir_gpio symbol has been included in the ivtv-0.4 release. I will try and update the howto/patch to reflect the change tonight (I don’t have any time at the moment to do so, unfortunately I have to work!). As it’s just those two hunks which are already applied, there should be no issue compiling it. Sorry about this, things are still moving pretty quickly with ivtv — hopefully it should settle down now there is a release. If you have any luck with it do let me know and I’ll push for the i2c patch to be included in the official release as well (somewhat simplifying the whole procedure!).

    Thanks,

    Mark

  5. Daniel Sherwood says:

    Mark

    We had a brief discussion a few months ago about support for the SKY digibox. At the time Hauppage didn’t support this STB so there was nothing I could do to get it working, however I notice that they have since released a new version of the IR Blatser software with support for new boxes including all of the PACA BSKYB boxes (codeset 0663). See http://www.hauppage.com/pages/support/blaster_table_update_1105.html & http://www.hauppage.com/pages/support/support_pvr150.html.

    Any chance you can re generate your codesets with this new vesrion so those of us with the newly supported boxes can get in control.

    Cheers in advance

    Daniel

  6. mark says:

    Daniel, I’ve now done this (and what a pain it was! Damn hauppauge and their closed source software!). You’ll need to download haup-ir-blaster.bin, send_power_new and lircd.conf again. Just follow the instructions in the other post. Hopefully it should work — my NTL pace box responds to some of the new codesets. Do let me know the outcome.

    Mark

  7. Daniel Sherwood says:

    Cheers Mark

    I’ll give it a go over Christmas and let you know how it goes.

    Daniel

  8. Brett Swaim says:

    I have been getting this compilation error after applying your patch (which did get me one step farther)

    loading cache ./config.cache
    checking for a BSD compatible install… (cached) /usr/bin/install -c
    checking whether build environment is sane… yes
    checking for mawk… (cached) gawk
    checking whether make sets ${MAKE}… (cached) yes
    checking for gcc… (cached) gcc
    checking whether the C compiler (gcc -O2 -g -Wall ) works… yes
    checking whether the C compiler (gcc -O2 -g -Wall ) is a cross-compiler… no
    checking whether we are using GNU C… (cached) yes
    checking whether gcc accepts -g… (cached) yes
    checking for style of include used by make… GNU
    checking dependency style of gcc… (cached) gcc3
    checking for a BSD compatible install… /usr/bin/install -c
    checking whether make sets ${MAKE}… (cached) yes
    checking for mknod… (cached) /usr/bin/mknod
    checking for mkfifo… (cached) /usr/bin/mkfifo
    checking for depmod… (cached) /sbin/depmod
    checking for libusb-config… no
    checking whether ln -s works… (cached) yes
    checking for Cygwin environment… (cached) no
    checking for mingw32 environment… (cached) no
    checking how to run the C preprocessor… (cached) gcc -E
    checking host system type… i686-pc-linux-gnu
    checking build system type… i686-pc-linux-gnu
    checking for ld used by GCC… (cached) /usr/i686-pc-linux-gnu/bin/ld
    checking if the linker (/usr/i686-pc-linux-gnu/bin/ld) is GNU ld… (cached) yes
    checking for /usr/i686-pc-linux-gnu/bin/ld option to reload object files… (cached) -r
    checking for BSD-compatible nm… (cached) /usr/bin/nm -B
    checking for a sed that does not truncate output… (cached) /usr/bin/sed
    checking how to recognise dependent libraries… (cached) pass_all
    checking for object suffix… (cached) o
    checking for executable suffix… (cached) no
    checking command to parse /usr/bin/nm -B output… (cached) ok
    checking for dlfcn.h… (cached) yes
    checking for ranlib… (cached) ranlib
    checking for strip… (cached) strip
    checking for objdir… .libs
    checking for gcc option to produce PIC… (cached) -fPIC
    checking if gcc PIC flag -fPIC works… (cached) yes
    checking if gcc static flag -static works… (cached) yes
    checking if gcc supports -c -o file.o… (cached) yes
    checking if gcc supports -c -o file.lo… (cached) yes
    checking if gcc supports -fno-rtti -fno-exceptions… yes
    checking whether the linker (/usr/i686-pc-linux-gnu/bin/ld) supports shared libraries… yes
    checking how to hardcode library paths into programs… immediate
    checking whether stripping libraries is possible… yes
    checking dynamic linker characteristics… GNU/Linux ld.so
    checking if libtool supports shared libraries… yes
    checking whether to build shared libraries… yes
    checking whether to build static libraries… yes
    checking whether -lc should be explicitly linked in… (cached) no
    creating libtool
    checking for ANSI C header files… (cached) yes
    checking whether time.h and sys/time.h may both be included… (cached) yes
    checking for fcntl.h… (cached) yes
    checking for limits.h… (cached) yes
    checking for sys/ioctl.h… (cached) yes
    checking for sys/time.h… (cached) yes
    checking for syslog.h… (cached) yes
    checking for unistd.h… (cached) yes
    checking for working const… (cached) yes
    checking for inline… (cached) inline
    checking for off_t… (cached) yes
    checking for pid_t… (cached) yes
    checking for size_t… (cached) yes
    checking whether struct tm is in sys/time.h or time.h… (cached) time.h
    checking return type of signal handlers… (cached) void
    checking for vprintf… (cached) yes
    checking for gethostname… (cached) yes
    checking for gettimeofday… (cached) yes
    checking for mkfifo… (cached) yes
    checking for select… (cached) yes
    checking for socket… (cached) yes
    checking for strdup… (cached) yes
    checking for strerror… (cached) yes
    checking for strtoul… (cached) yes
    checking for snprintf… (cached) yes
    checking for strsep… (cached) yes
    checking for vsyslog… (cached) yes
    checking for daemon… (cached) yes
    checking for forkpty… (cached) no
    checking for forkpty in -lutil… (cached) yes
    checking for vga.h… (cached) no
    checking for X… (cached) libraries , headers
    checking for dnet_ntoa in -ldnet… (cached) no
    checking for dnet_ntoa in -ldnet_stub… (cached) no
    checking for gethostbyname… (cached) yes
    checking for connect… (cached) yes
    checking for remove… (cached) yes
    checking for shmat… (cached) yes
    checking for IceConnectionNumber in -lICE… (cached) yes
    checking for getopt_long… (cached) yes
    checking for mktemp… (cached) yes
    checking for Linux kernel sources… (cached) /lib/modules/2.6.14-gentoo-r4/build/
    checking for which drivers can be installed on this system…
    checking for caraca_init in -lcaraca_client… (cached) no
    checking for ir_strerror in -lirman… (cached) no
    checking for ir_strerror in -lirman_sw… (cached) no
    checking for portaudio.h… (cached) no
    checking for alsa/asoundlib.h… (cached) yes
    checking for snd_pcm_open in -lasound… (cached) yes
    checking for ALSA SB RC hwdep support… yes
    checking for scsi/sg.h… (cached) yes
    checking for linux/input.h… (cached) yes
    checking for sys/soundcard.h… (cached) yes
    updating cache ./config.cache
    creating ./config.status
    creating Makefile
    creating drivers/Makefile
    creating drivers/lirc_atiusb/Makefile
    creating drivers/lirc_bt829/Makefile
    creating drivers/lirc_cmdir/Makefile
    creating drivers/lirc_dev/Makefile
    creating drivers/lirc_gpio/Makefile
    creating drivers/lirc_i2c/Makefile
    creating drivers/lirc_igorplugusb/Makefile
    creating drivers/lirc_imon/Makefile
    creating drivers/lirc_it87/Makefile
    creating drivers/lirc_mceusb/Makefile
    creating drivers/lirc_mceusb2/Makefile
    creating drivers/lirc_parallel/Makefile
    creating drivers/lirc_sasem/Makefile
    creating drivers/lirc_serial/Makefile
    creating drivers/lirc_sir/Makefile
    creating drivers/lirc_streamzap/Makefile
    creating daemons/Makefile
    creating tools/Makefile
    creating doc/Makefile
    creating doc/man/Makefile
    creating config.h
    config.h is unchanged

    You will have to use the lirc_i2c kernel module.

    Now enter ‘make’ and ‘make install’ to compile and install the package.

    Making install in drivers
    make[1]: Entering directory `/var/user_installed/lirc-0.7.2/drivers’
    Making install in lirc_dev
    make[2]: Entering directory `/var/user_installed/lirc-0.7.2/drivers/lirc_dev’
    make[3]: Entering directory `/var/user_installed/lirc-0.7.2/drivers/lirc_dev’
    test -c /dev/lirc || (/bin/sh ../../mkinstalldirs /dev && /usr/bin/mknod /dev/lirc c 61 0)
    /bin/sh ../../mkinstalldirs /lib/modules/2.6.14-gentoo-r4/misc
    /usr/bin/install -c -m 644 lirc_dev.ko /lib/modules/2.6.14-gentoo-r4/misc/lirc_dev.ko
    /sbin/depmod -a
    make[3]: Leaving directory `/var/user_installed/lirc-0.7.2/drivers/lirc_dev’
    make[2]: Leaving directory `/var/user_installed/lirc-0.7.2/drivers/lirc_dev’
    Making install in lirc_i2c
    make[2]: Entering directory `/var/user_installed/lirc-0.7.2/drivers/lirc_i2c’
    mv Makefile Makefile.automake
    cp ../Makefile.kernel Makefile
    make -C /lib/modules/2.6.14-gentoo-r4/build/ SUBDIRS=/var/user_installed/lirc-0.7.2/drivers/lirc_i2c modules \
    KBUILD_VERBOSE=1
    make[3]: Entering directory `/usr/src/linux-2.6.14-gentoo-r4′
    mkdir -p /var/user_installed/lirc-0.7.2/drivers/lirc_i2c/.tmp_versions
    make -f scripts/Makefile.build obj=/var/user_installed/lirc-0.7.2/drivers/lirc_i2c
    gcc -m32 -Wp,-MD,/var/user_installed/lirc-0.7.2/drivers/lirc_i2c/.lirc_i2c.o.d -nostdinc -isystem /usr/lib/gcc-lib/i686-pc-linux-gnu/3.3.5-20050130/include -D__KERNEL__ -Iinclude -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -ffreestanding -O2 -fomit-frame-pointer -pipe -msoft-float -mpreferred-stack-boundary=2 -march=i686 -Iinclude/asm-i386/mach-default -DIRCTL_DEV_MAJOR=61 -DEXPORT_SYMTAB -DHAVE_CONFIG_H -I. -I. -I../.. -I /var/user_installed/lirc-0.7.2/drivers/lirc_i2c/../.. -I /lib/modules/2.6.14-gentoo-r4/build//include/ -DMODULE -DKBUILD_BASENAME=lirc_i2c -DKBUILD_MODNAME=lirc_i2c -c -o /var/user_installed/lirc-0.7.2/drivers/lirc_i2c/lirc_i2c.o /var/user_installed/lirc-0.7.2/drivers/lirc_i2c/lirc_i2c.c
    In file included from /var/user_installed/lirc-0.7.2/drivers/lirc_i2c/lirc_i2c.c:57:
    /var/user_installed/lirc-0.7.2/drivers/kcompat.h:173:19: warning: extra tokens at end of #ifndef directive
    In file included from include/linux/rcuref.h:36,
    from include/linux/fs.h:12,
    from /var/user_installed/lirc-0.7.2/drivers/lirc_dev/lirc_dev.h:24,
    from /var/user_installed/lirc-0.7.2/drivers/lirc_i2c/lirc_i2c.c:58:
    include/linux/interrupt.h:30: error: conflicting types for `irqreturn_t’
    /var/user_installed/lirc-0.7.2/drivers/kcompat.h:166: error: previous declaration of `irqreturn_t’
    In file included from include/linux/rcuref.h:36,
    from include/linux/fs.h:12,
    from /var/user_installed/lirc-0.7.2/drivers/lirc_dev/lirc_dev.h:24,
    from /var/user_installed/lirc-0.7.2/drivers/lirc_i2c/lirc_i2c.c:58:
    include/linux/interrupt.h:32:1: warning: “IRQ_NONE” redefined
    In file included from /var/user_installed/lirc-0.7.2/drivers/lirc_i2c/lirc_i2c.c:57:
    /var/user_installed/lirc-0.7.2/drivers/kcompat.h:168:1: warning: this is the location of the previous definition
    In file included from include/linux/rcuref.h:36,
    from include/linux/fs.h:12,
    from /var/user_installed/lirc-0.7.2/drivers/lirc_dev/lirc_dev.h:24,
    from /var/user_installed/lirc-0.7.2/drivers/lirc_i2c/lirc_i2c.c:58:
    include/linux/interrupt.h:33:1: warning: “IRQ_HANDLED” redefined
    In file included from /var/user_installed/lirc-0.7.2/drivers/lirc_i2c/lirc_i2c.c:57:
    /var/user_installed/lirc-0.7.2/drivers/kcompat.h:171:1: warning: this is the location of the previous definition
    In file included from include/linux/rcuref.h:36,
    from include/linux/fs.h:12,
    from /var/user_installed/lirc-0.7.2/drivers/lirc_dev/lirc_dev.h:24,
    from /var/user_installed/lirc-0.7.2/drivers/lirc_i2c/lirc_i2c.c:58:
    include/linux/interrupt.h:34:1: warning: “IRQ_RETVAL” redefined
    In file included from /var/user_installed/lirc-0.7.2/drivers/lirc_i2c/lirc_i2c.c:57:
    /var/user_installed/lirc-0.7.2/drivers/kcompat.h:174:1: warning: this is the location of the previous definition
    make[4]: *** [/var/user_installed/lirc-0.7.2/drivers/lirc_i2c/lirc_i2c.o] Error 1
    make[3]: *** [_module_/var/user_installed/lirc-0.7.2/drivers/lirc_i2c] Error 2
    make[3]: Leaving directory `/usr/src/linux-2.6.14-gentoo-r4′
    make[2]: *** [lirc_i2c.o] Error 2
    make[2]: Leaving directory `/var/user_installed/lirc-0.7.2/drivers/lirc_i2c’
    make[1]: *** [install-recursive] Error 1
    make[1]: Leaving directory `/var/user_installed/lirc-0.7.2/drivers’
    make: *** [install-recursive] Error 1

    Just to let you know, I fixed this by removing line 166 of drivers/kcompat.h. Compiles fine now.

  9. Brett Swaim says:

    But alas, irw still gives no output when I try to use my remote plugged into my pvr-250…

  10. Brett Swaim says:

    but when I cat /dev/lirc it gives some crazy data at just the same time I press the button… getting warmer!!

  11. Michael Best says:

    Here’s what I did under KnoppMyth R5A26.

    # svn co http://ivtvdriver.org/svn/ivtv/tags/0.4.0
    # cd 0.4.0/driver
    # wget -O i2c-0.4.0.patch http://ivtvdriver.org/trac/attachment/ticket/29/i2c-0.4.0.patch?format=raw
    # patch -p0

  12. Michael Best says:

    Looks like you can’t put certain code in comments.

    See this for the rest:
    http://mysettopbox.tv/phpBB2/viewtopic.php?p=43927#43927

  13. mark says:

    Brett — there’s no need to use any of this stuff with a PVR-250. That has a completely different IR receiver chip. I’ve never done any work with it, and equally I’ve never seen any reports of IR problems. If you are having trouble, I would recommend posting to the ivtv or lirc lists.

    Michael — what’s the actual issue? I can’t see a specific problem from what you have written.

    Thanks,

    Mark

  14. Patrick says:

    Mark,

    I ran through your how-to and got stuck. Upon modprobing lirc_dev and lirc_pvr150, I receive a “FATAL: Module lirc_i2c not found.” error.

    dmesg output shows:

    lirc_dev: module successfully unloaded
    lirc_dev: IR Remote Control driver registered, at major 61
    Linux video capture interface: v1.00
    ivtv: ==================== START INIT IVTV ====================
    ivtv: version 0.4.1 (tagged release) loading
    ivtv: Linux version: 2.6.11-gentoo-r9 SMP PENTIUMIII gcc-3.3
    ivtv: In case of problems please include the debug info between
    ivtv: the START INIT IVTV and END INIT IVTV lines, along with
    ivtv: any module options, when mailing the ivtv-users mailinglist.
    ivtv0: Autodetected WinTV PVR 150 card (cx23416 based)
    ivtv0: i2c attach to card #0 ok [client=eeprom, addr=50]
    ivtv0 warning: i2c client addr: 0x50 not found for command 0x0!
    ivtv0: Error -19 reading Hauppauge eeprom.
    ivtv0: Possible causes: the tveeprom module was not loaded, or
    ivtv0: the eeprom kernel module was loaded before the tveeprom module.
    tuner (ivtv): chip found at addr 0xc2 i2c-bus ivtv i2c driver #0
    ivtv0: i2c attach to card #0 ok [client=(tuner unset), addr=61]
    cx25840: Unknown parameter `i2c_enable’
    ivtv0: Failed to load module cx25840
    ivtv0 warning: i2c client addr: 0x44 not found for command 0xc008561c!
    wm8775 2-001b: ivtv driver
    wm8775 2-001b: chip found @ 0x36 (ivtv i2c driver #0)
    ivtv0: i2c attach to card #0 ok [client=wm8775, addr=1b]
    ivtv0: Could not detect tuner standard, defaulting to NTSC.
    ivtv0: unable to open firmware v4l-cx2341x-enc.fw
    ivtv0: did you put the firmware in the hotplug firmware directory?
    ivtv0 warning: failed loading encoder firmware
    ivtv0 warning: Error loading firmware -3!
    ivtv0: Error -3 initializing firmware.
    ivtv0: Error -12 on initialization
    ivtv: probe of 0000:02:07.0 failed with error -12
    ivtv: ==================== END INIT IVTV ====================

    It looks like, even though I patched lirc-0.7.2 (used your pre-patched source), ivtv is still looking for the lirc_i2c driver.

    Any advice?

    Thanks,
    -Patrick

  15. Auro Kioshi Takara says:

    Hi! I’m using SuSE 10.0 and and the prepatched lirc-0.7.2 does not compile under newer kernels.
    Is it possible to apply the patch in lirc-0.8.0pre3?

  16. SteveH says:

    Firstly, great job on this driver and helping people getting the PVR-150 remote working!

    I’m almost there. I have lirc 0.7.2 with the patch and ivtv 0.4.0 with the patch. The chip is detected (dmesg reports i2c attach to card #0 ok for the PVR150 IR RX and TX, addr 71 and 70).

    The problem is when I reboot the machine. The chip is still detected but irw doesn’t report any IR signals. ivtvctrl -reset-ir doesn’t help, and neither does a cold reboot. The only way I can get the remote to work again is to boot into windows where it works fine and reboot into linux. It then works again (irw shows remote button presses) until the machine is rebooted (warm or cold).

    Is it possible the windows driver is doing some sort of necessary initialization? Possibly to the firmware on the card?

    If necessary I can trace through code to track this down.

    I’m running Knoppmyth R5A26.

  17. SteveH says:

    If it is of any help, this problem on reboot doesn’t occur when the lirc_i2c driver from lirc-0.7.2 is used.

  18. SteveH says:

    Sorry, one more piece of information. After reboot when the IR receiver doesn’t work, the transmitter does continue to work. send_power_new causes the light on the transmitter to flash.

  19. mark says:

    Try adding:

    ir->shutdown = 0;

    at around lirc_pvr150.c:1211 (in ir_attach).

    I think it is shutting down the polling thread immediately (you should be able to see if that is the case from a log with the module option debug=1), hence nothing is received. I have seen this once before, but I haven’t gotten around to updating anything yet.

  20. SteveH says:

    Excellent, thank you! That has fixed the problem.

    If you update the code and need me to do any testing let me know, I can reproduce it reliably.

Leave a Reply

Your email address will not be published. Required fields are marked *