Reference / Paper · 2015
Imperceptible complementary positioning targets flashed in the displayed image for a camera pointer.
US008982050B2 (12) United States Patent (10) Patent N0.: (45) Date of Patent: Solomon et a]. (54) (56) MOTION COMPENSATION IN AN US 8,982,050 B2 b1ar.17,2015 References Cited INTERACTIVE DISPLAY SYSTEM U.S. PATENT DOCUMENTS (71) Applicant: Interphase Corporation, Plano, TX (Us) (72) Inventors: Yoram Solomon, Plano, TX (US); 5,982,352 A 11/1999 Pryor 6,133,906 A * 10/2000 6,995,748 B2 2/2006 Gordon et al. (Continued) Branislav Kisacanin, Plano, TX (US) (73) Assignee: Interphase Corporation, Carrollton, TX (Us) (*) Notice: FOREIGN PATENT DOCUMENTS WO 2013054155 Al Subject to any disclaimer, the term of this patent is extended or adjusted under 35 U.S.C. 154(b) by 47 days. Geaghan ..................... .. 345/179 4/2013 OTHER PUBLICATIONS International Search Report and Written Opinion of the ISA, PCT International Application PCT/US2013/065407 (Jan. 28, 2014). (21) Appl. No.: 14/056,286 (22) Filed: (Continued) Oct. 17, 2013 (65) Primary Examiner * Kimnhung Nguyen (74) Attorney, Agent, or Firm * Anderson & Levine, L.L.P. Prior Publication Data US 2014/0111433 A1 Apr. 24, 2014 (57) Related US. Application Data (60) Provisional application No. 61/716,308, ?led on Oct. 19, 2012, provisional application No. 61/718,985, ?led on Oct. 26, 2012. (51) Int. Cl. G06F 3/033 G09G 5/08 US. Cl. (2013.01); G06F 3/0317 (2013.01) USPC ......... .. 345/158; 345/156; 345/173; 345/180; 348/63 Field of Classi?cation Search CPC device, and positioning circuitry capable of determining absolute and relative positions of the display at which the pointing device is aimed. The pointing device captures images displayed by the computer, including one or more targets are presented as patterned modulation of the intensity (e.g., variation in pixel intensity) in a display frame of the visual payload, followed by the opposite modulation in a successive frame. At least two captured image frames are CPC .......... .. G06F 3/0346 (2013.01); G06F 3/0304 (58) An interactive display system including a wireless pointing human-imperceptible positioning targets. The positioning (2013.01) (2006.01) (Continued) (52) ABSTRACT G06F 3/03393; G06F 3/042; G06F 3/038; G06F 1/1626; G06F 3/0304 subtracted from one another to recover the positioning target in the captured visual data and to remove the displayed image payload. Motion of the pointing device between the two frames is detected by relative motion sensors, and used to align the positions of the positioning targets in the captured images for those frames to improve the ?delity of the recov ered positioning target following the subtraction. USPC ................... .. 345/1564158, 1734183; 348/63 See application ?le for complete search history. 20 Claims, 11 Drawing Sheets SAMPLE mono" SENSORS ATA SAMPLE TIME I DISPLAYING COMBINED PAYLOAD IMAGE DATAAND POSITIONING TARGETS 42 DETERMINE CHANGE IN POSITION RELATIVE TO A PRIOR SAMPLE TIME CAP URE IMAGE CALCULATE RELATIVE MOTION BETWEEN FRAME DATA FRAME TIMES] AND j+1 NEXT FRAME TIME SHIFT IMAGE FRAME DATA OF ONE OF FRAMES 1. 1+1 BY CALCULATED RELATIVE MOTION SUBTRACT SHIFTED IMAGE FRAME DATA OF FRAMES],|+1 IDENTIFY LOCATION OF POSITIONING TARGET PATTERN IN DIFFERENCE FRAME IMAGE DATA COMPUTE ABSOLUTE POSITION AT DISPLAY r46 US 8,982,050 B2 Page 2 (51) Int. Cl. 2005/0260986 A1 11/2005 Sun et a1. G061” “346 (201301) 588218132211 2‘1 12/588; Efeketiil' G06F 3/03 (200601) 2008/0275667 A1 err et _ (56) References Clted U.S. PATENT DOCUMENTS . 11/2008 011m 2009/0002344 A1 1/2009 Wilson et a1. 2010/0085325 A1* 4/2010 King-Smith etal. ....... .. 345/174 2010/0228632 A1* 9/2010 Rodriguez ........ .. 2011/0227827 A1* 9/2011 Solomon etal. ............ .. 345/158 . 705/1466 7,158,118 B2 7,187,412 B1 1/2007 Liberty 3/2007 Silverstein 2011/0298710 A1 2012/0038549 A1 2012/0086725 A1 12/2011 Ruckhaeberle etal. 2/2012 Mandella et a1. 4/2012 Joseph et a1. 7,233,316 B2 8,184,166 B2 8,217,997 B2 6/2007 Smith et a1. 5/2012 Wang etal. 7/2012 Solomon et a1. 2013/0055143 A1* 2013/0063537 A1* 2014/0118496 A1* 2/2013 Martin et a1. ............... .. 715/779 3/2013 Emori ...................... .. 348/1401 5/2014 Calaklietal. ................. .. 348/46 8,446,364 B2 8,451,223 B2 5/2013 Solomon 5/2013 Choietal. 8,456,534 B2 6/2013 Henty 8,471,812 B2 6/2013 Bunch 8,631,208 132* 3/2014 Yoshino ,,,,,,,,,,,,,,,,,,,,,,,, ,, 348/65 8,896,741 B2* 11/2014 Akagietal. 2002/0075250 A1* OTHER PUBLICATIONS U~S~APPI~N0~ 14/018,695r?led 5695,2013 348/333.12 6/2002 Shigeta etal. .............. .. 345/204 * cited by examiner US. Patent Mar. 17, 2015 Sheet 1 0f 11 FIG. 1a STRATEGIC PLAN US 8,982,050 B2 '/ US. Patent Mar. 17, 2015 Sheet 2 0f 11 US 8,982,050 B2 FIG. 1b 2" STRATEGIC PLAN —/ E —J I DEE: DOI SPKR 24/ RECEIVER FIG. 5 l— I 36 _ _ _ _ _ _ _ _ _ 135 _ _ _ _ | I| : ABSOLUTE POSITIONING SUBSYSTEM VISUAL SENSORS MOTION SENSORS _ r_______| 3_7 | I A I SHFT | —I—> RELATIVE POSITIONING SUBSYSTEM | l | | _l | I| ABS US. Patent Mar. 17, 2015 |_________| Sheet 3 0f 11 US 8,982,050 B2 US. Patent Mar. 17, 2015 Sheet 5 0f 11 US 8,982,050 B2 FIG. 3d US. Patent Mar. 17, 2015 Sheet 6 0f 11 US 8,982,050 B2 US. Patent Mar. 17, 2015 Sheet 7 0f 11 DISPLAYING COMBINED 41 \ PAYLOAD IMAGE DATA AND POSITIONING TARGETS US 8,982,050 B2 SAMPLE MOTION SENSORS AT A SAMPLE TIME / 46 DETERMINE CHANGE IN 48 POSITION RELATIVE TO A PRIOR SAMPLE TIME / II 42 \ II CALCULATE RELATIVE CAPTURE IMAGE FRAME DATA MOTION BETWEEN FRAME TIMES j AND j+1 NEXT FRAME TIME II SHIFT IMAGE FRAME DATA OF ONE OF FRAMES 52 f I, j+1 BY CALCULATED RELATIVE MOTION I SUBTRACT SHIFTED IMAGE 54 / FRAME DATA OF FRAMESj, j+1 I 56 / IDENTIFY LOCATION OF POSITIONING TARGET PATTERN IN DIFFERENCE FRAME IMAGE DATA I 58 / COMPUTE ABSOLUTE POSITION AT DISPLAY FIG. 6 / 50 US. Patent Mar. 17, 2015 Sheet 8 0f 11 US 8,982,050 B2 F \ 70\ 20\ y x J Z X 10 V FIG. 7 25' I. ________________ ___/_______I I VISUAL SENSORS 35 — --q ' g i : i I SIZE I l MOT'ON SENSORS E ABS | I = ABSOLUTE POSITIONING SUBSYSTEM | SHAPE II X_Y_Z I R_P_Y A l = I I +S_H_F_T__ SENSOR TSHFT SUBSYSTEM I 40 | II FUSION l _ LJIF> RELATIVE POSITIONING SUBSYSTEM i REL = I = I Q I L _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ : :I FPOS _ l | _ I ' _ _ _ _ _ __J US. Patent Mar. 17, 2015 Sheet 9 0f 11 US 8,982,050 B2 II IDENTIFY SIMILAR ELEMENT(S) IN 4 ALTERNATING (ODD OR /6 EVEN) FRAMES I DETERMINE MOTION DISPLAYING COMBINED / 56 BETWEEN FRAMES] AND 1+2 41 \ PAYLOAD IMAGE DATA AND I POSITIONING TARGETS I "1ch (BY 1 0R 2) V v 42 \ INTERPOLATE To DERIVE CAPTURE IMAGE RELATIVE MOTION BETWEEN / 68 FRAME DATA FRAME TIMESj AND j+1 NEXT FRAME TIME II SHIFT IMAGE FRAME DATA OF ONE OF FRAMES 52 f j, j+1 BY CALCULATED RELATIVE MOTION I SUBTRACT SHIFTED IMAGE 54 / FRAME DATA OF FRAMES j, j+1 56 / IDENTIFY LOCATION OF POSITIONING TARGET IN DIFFERENCE FRAME IMAGE DATA 58 _/ COMPUTE ABSOLUTE POSITION AT DISPLAY I I FIG. 9 US. Patent Mar. 17, 2015 Sheet 10 0f 11 FIG.V 10b US 8,982,050 B2 US. Patent Mar. 17, 2015 Sheet 11 0f 11 US 8,982,050 B2 FROM 42 I 72 \ DETERMINE MOTION BETWEEN FRAMESj ANDj+2 V 74 \ DETERMINE MOTION BETWEEN NEXT ADJACENT PAIR OF FRAMES j+1 AND j+3 V ESTIMATE ANGLE OF MOTION AT FRAME j+2 FROM 76 / THE RELATIVE MOTION BETWEEN FRAMES j+1 ANDj+3 V INTERPOLATE TO DERIVE RELATIVE MOTION BETWEEN 68. / FRAMESjANDj+1 BY BACK INTERPOLATION FROM THE POSITION AT FRAME j+2 USING THE ANGLE OF MOTION I To 52 FIG. 10c FROM 42 I 30 \ RECEIVE IMAGE FRAME DATA FR0M SUCCESSIVE FRAMES j, j+1 V 82 \ INVERT IMAGE FRAME DATA FOR FRAME j+1 V 84 \ BALANCE BRIGHTNESS, COLOR, CONTRAST, ETC. OF INVERTED FRAME j+1 86 / IDENTIFY SIMILAR ELEMENT(S) IN SUCCESSIVE FRAMES j, j+1 V V DETERMINE MOTION 88 / BETWEEN FRAMES j AND j+1 I TO 52 FIG. 1 1 US 8,982,050 B2 1 2 MOTION COMPENSATION IN AN INTERACTIVE DISPLAY SYSTEM US. Pat. No. 8,217,997, issued Jul. 10, 2012, entitled “Interactive Display System”, commonly assigned herewith CROSS-REFERENCE TO RELATED APPLICATIONS and incorporated herein by reference, describes an interactive display system including a wireless human interface device (“HID”) constructed as a handheld pointing device including a camera or other video capture system. The pointing device captures images displayed by the computer, including one or more human-imperceptible positioning targets inserted by 2012; and Provisional Application No. 61/718,985, ?led Oct. 1 the computer into the displayed image data. The location, 0 size, and orientation of the recovered positioning target iden 26, 2012; both incorporated herein by this reference. tify the aiming point of the remote pointing device relative to This application claims priority, under 35 U.S.C. §119(e), of Provisional Application No. 61/716,308, ?led Oct. 19, the display. The positioning of the aiming point of the pointing device STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT according to the approach described in the above-referenced US. Pat. No. 8,217,997 is performed at a rate corresponding to the frame rate of the display system. More speci?cally, a Not applicable. new position can be determined as each new frame of data is BACKGROUND OF THE INVENTION displayed, by the combination of the new frame (and its positioning target) and the immediately previous frame (and its complementary positioning target). This approach works This invention is in the ?eld of interactive display systems. Embodiments of this invention are more speci?cally directed to the positioning of the location at a display to which a quite well in many situations, particularly in the context of navigating and controlling a graphical user interface in a control device is pointing during the interactive operation of computer system, such as pointing to and “clicking” icons, a computer system. click-and-drag operations involving displayed windows and The ability of a speaker to communicate a message to an 25 frames, and the like. A particular bene?t of this approach audience is generally enhanced by the use of visual informa described in US. Pat. No. 8,217,997, is that the positioning is tion, in combination with the spoken word. In the modern era, “absolute”, in the sense that the result of the determination is the use of computers and associated display systems to gen a speci?c position on the display (e. g., pixel coordinates). The erate and display visual information to audiences has become accuracy of the positioning carried out according to this commonplace, for example by way of applications such as the 30 approach is quite accurate over a wide range of distances POWERPOINT presentation software program available between the display and the handheld device, for example ranging from in physical contact with the display screen to from Microsoft Corporation. For large audiences, such as in tens of feet away. an auditorium environment, the display system is generally a Conventional human interface devices based on motion projection system (either front or rear proj ection). For smaller audiences such as in a conference room or classroom envi 35 sensors are also known in the art. Motion sensors sense special screen and thus are even more readily deployed, are 40 motion of the device over a sequence of sample times. Examples of motion sensors include inertial sensors such as accelerometers, gyroscopes, magnetic ?eld sensors such as magnetometers, and visual systems such as those used in optical mice. The positioning result based on motion sensors ronment, ?at-panel (e.g., liquid crystal) displays have become popular, especially as the cost of these displays has fallen over recent years. New display technologies, such as small projectors (“pico-projectors”), which do not require a now reaching the market. For presentations to very small is relative, in the sense that an absolute position of the display audiences (e.g., one or two people), the graphics display of a is not directly determined, but rather the motion sensors deter mine the position and attitude of the device, and from that the laptop computer may suf?ce to present the visual informa tion. In any case, the combination of increasing computer pointed-to location, relative to that at a previous point in time. power and better and larger displays, all at less cost, has 45 However, the sample rate at which motion sensor-based pointing devices operate is not limited by the frame rate of the increased the use of computer-based presentation systems, in a wide array of contexts (e.g., business, educational, legal, entertainment). A typical computer-based presentation involves the display, and can be much higher, assuming proper registration of the relative positioning. In addition, fewer computations are required to derive the relative positioning result, as com speaker standing remotely from the display system, so as not 50 pared with those required for absolute positioning. Unfortu to block the audience’s view of the visual information. nately, however, because the positioning provided by these Because the visual presentation is computer-generated and devices is relative, drift or other error can accumulate over computer-controlled, the presentation is capable of being time. Error is exacerbated for those devices relying on accel erometer motion sensing, as two integrations are required in order to convert sensed accelerations into linear distances. As such, the accuracy of relative positioning based on motion sensors is generally inferior to that of absolute positioning interactively controlled, to allow selection of visual content of particular importance to a speci?c audience, annotation or 55 illustration of the visual information by the speaker during the presentation, and invocation of effects such as zooming, selecting links to information elsewhere in the presentation approaches. (or online), moving display elements from one display loca Copending US. patent application Ser. No. 14/018,695, tion to another, and the like. This interactivity greatly 60 ?led Sep. 5, 2013, commonly assigned herewith and incor enhances the presentation, making it more interesting and engaging to the audience. The ability of a speaker to interact, from a distance, with displayed visual content, is therefore desirable. More speci? porated herein by this reference, describes an interactive dis play system including a wireless pointing device and posi tioning circuitry capable of determining both absolute and relative positions of the display at which the pointing device cally, a hand-held device that a remotely-positioned operator 65 is aimed. A comparison between the absolute and relative positions at a given time is used to compensate the relative could use to point to, and interact with, the displayed visual information is therefore desirable. position determined by the motion sensors, enabling both US 8,982,050 B2 3 4 rapid and frequent positioning provided by the motion sen FIG. 9 is a ?ow diagram illustrating the operation of the sors and also the excellent accuracy provided by absolute architecture of FIG. 4 according to an alternative embodiment of the invention. FIGS. 10a and 10b are plots illustrating the determination of a position at an image capture time for a frame, based on relative motion sensed between frames on either side of that frame. FIG. 100 is a ?ow diagram illustrating the operation of the positioning. BRIEF SUMMARY OF THE INVENTION Embodiments of this invention provide a system and method for rapidly and accurately determining an absolute position of the location at a display at which a handheld relative positioning subsystem in the architecture of FIG. 4 human interface device is pointing during the operation of an according to an alternative implementation of an embodiment of the invention. FIG. 11 is a ?ow diagram illustrating the operation of the interactive display system. Embodiments of this invention provide such a system and method in which such absolute positioning can be performed in systems using visual information acquired at the frame rate of the display system, even in situations in which the user is relative positioning subsystem in the architecture of FIG. 4 according to another alternative implementation of an moving the pointing device rapidly. embodiment of the invention. Other objects and advantages of embodiments of this invention will be apparent to those of ordinary skill in the art DETAILED DESCRIPTION OF THE INVENTION having reference to the following speci?cation together with its drawings. This invention will be described in connection with one or Embodiments of this invention may be implemented into 20 more of its embodiments, namely as implemented into a computerized presentation system including a display visible an interactive display system and method of operating the by an audience, as it is contemplated that this invention will same in which a remote human interface device includes an be particularly bene?cial when applied to such a system. image capture subsystem for identifying an absolute location However, it is also contemplated that this invention can be at the display at which the device is pointing, and also one or more motion sensors for detecting relative motion of the 25 useful in connection with other applications, such as gaming systems, general input by a user into a computer system, and pointing device. Absolute positioning is based on the sensing the like. Accordingly, it is to be understood that the following of complementary positioning targets displayed in ?rst and description is provided by way of example only, and is not second frames; subtraction of the frame data from the ?rst and intended to limit the true scope of this invention as claimed. second frames recovers the positioning target pattern while FIG. 1a illustrates a simpli?ed example of an environment canceling out the human-visible content. Inter-frame relative 30 in which embodiments of this invention are useful. As shown motion of the pointing device, as sensed by the motion sen sors, compensates the position of the positioning target in the in FIG. 1a, speaker SPKR is giving a live presentation to later one of the frames to align the positioning targets in the audience A, with the use of visual aids. In this case, the visual aids are in the form of computer graphics and text, generated ?rst and second frames, facilitating the absolute positioning determination. 35 by computer 22 and displayed on room-size graphics display 20, in a manner visible to audience A. As known in the art, BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING such presentations are common in the business, educational, entertainment, and other contexts, with the particular audi ence size and system elements varying widely. The simpli?ed FIGS. 1a and 1b are schematic perspective views of a 40 example of FIG. 1a illustrates a business environment in which audienceA includes several or more members viewing speaker presentation being carried out using an interactive display system according to embodiments of the invention. the presentation; of course, the size of the environment may FIGS. 2a and 2b are electrical diagrams, in block form, vary from an auditorium, seating hundreds of audience mem bers, to a single desk or table in which audience A consists of each illustrating an interactive display system according to an embodiment of the invention. 45 a single person. FIGS. 3a through 3d are views of a display illustrating the The types of display 20 used for presenting the visual aids to audienceA can also vary, often depending on the size of the operation of a visual absolute positioning system in which presentation environment. In rooms ranging from conference embodiments of the invention may be implemented. FIGS. 4a through 4d are views of a display illustrating the rooms to large-scale auditoriums, display 20 may be a pro operation of a visual absolute positioning system in the con 50 jection display, including a projector disposed either in front text of relative motion, and in which embodiments of the of or behind a display screen. In that environment, computer 22 would generate the visual aid image data and forward it to invention may be implemented. FIG. 5 is a functional diagram, in block form, illustrating the projector. In smaller environments, display 20 may be an the functional architecture of the positioning subsystems in external ?at-panel display, such as of the plasma or liquid an interactive display system according to embodiments of 55 crystal (LCD) type, directly driven by a graphics adapter in the invention. computer 22. For presentations to one or two audience mem FIG. 6 is a ?ow diagram illustrating the operation of the bers, computer 22 in the form of a laptop or desktop computer architecture of FIG. 4 according to embodiments of the inven may simply use its own display 20 to present the visual tion. information. Also for smaller audiences A, hand-held projec FIG. 7 is a perspective view of the orientation axes of a 60 tors (e. g., “pocket projectors” or “pico projectors”) are pointing device as used in the cooperation between absolute becoming more common, in which case the display screen and relative positioning according to an embodiment of the may be a wall or white board. invention. The use of computer presentation software to generate and present graphics and text in the context of a presentation is FIG. 8 is a functional diagram, in block form, illustrating the functional architecture of the positioning subsystems in 65 now commonplace. A well-known example of such presen tation software is the POWERPOINT software program an interactive display system according to an alternative embodiment of the invention. available from Microsoft Corporation. In the environment of US 8,982,050 B2 5 6 FIG. 1a, such presentation software will be executed by com puter 22, with each slide in the presentation displayed on display 20 as shown in this example. Of course, the particular embodiments of this invention. To do so, computer 22 coop erates with positioning circuitry 25, which determines the position of display screen 20 to which pointing device 10 is visual information need not be a previously created presen tation executing at computer 22, but instead may be a web pointing. As will become apparent from the following description, this positioning determination is based on point page accessed via computer 22; a desktop display including ing device 10 detecting one or more positioning targets dis played at display screen 20. icons, program windows, and action buttons; video or movie content from a DVD or other storage device being read by computer 22. Other types of visual information useful in connection with embodiments of this invention will be appar ent to those skilled in the art having reference to this speci? cation. In its payload image generation function, computer 22 will generate or have access to the visual information to be dis played (i.e., the visual “payload” images), for example in the form of a previously generated presentation ?le stored in memory, or in the form of active content such as computer 22 In FIG. 1a, speaker SPKR is standing away from display may retrieve over a network or the Internet; for a “white 20, so as not to block the view of audienceA and also to better boar ” application, the payload images will include the inputs provided by the user via pointing device 10, typically dis played on a blank background. This human-visible payload image frame data from computer 22 will be combined with engage audience A. According to embodiments of this inven tion, speaker SPKR uses a handheld human interface device (HID), in the form of pointing device 10, to remotely interact with the visual content displayed by computer 22 at display positioning target image content generated by target genera 20. This interactive use of visual information displayed by tor function 23 that, when displayed at graphics display 20, display 20 provides speaker SPKR with the ability to extem 20 can be captured by pointing device 10 andused by positioning circuitry 25 to deduce the location pointed to by pointing porize the presentation as deemed useful with a particular audience A, to interface with active content (e.g., Internet device 10. Graphics adapter 27 includes the appropriate func tionality suitable for presenting a sequence of frames of links, active icons, virtual buttons, streaming video, and the like), and to actuate advanced graphics and control of the image data, including the combination of the payload image presentation, without requiring speaker SPKR to be seated at 25 data and the positioning target image content, in the suitable or otherwise “pinned” to computer 22. display format, to projector 21. Projector 21 in turn projects FIG. 1b illustrates another use of the system and method of the corresponding images I at display screen 20, in this pro jection example. embodiments of this invention, in which speaker SPKR closely approaches display 20 to interact with the visual con The particular construction of computer 22, positioning tent. In this example, display 20 is operating as a “white 30 circuitry 25, target generator circuitry 23, and graphics board” on which speaker SPKR may “draw” or “write” using adapter 27 can vary widely. For example, it is contemplated pointing device 10 to actively draw content as annotations to that a single personal computer or workstation (in desktop, the displayed content, or even on a blank screen as suggested laptop, or other suitable form), including the appropriate pro by FIG. 1b. Typically, this “drawing” and “writing” would be cessing circuitry (CPU, or microprocessor) and memory, can carried out while placing pointing device 10 in actual physical 35 be constructed and programmed to perform the functions of contact with, or at least in close proximity to, display 20. The generating the payload images, generating the positioning hardware, including display 20, in the application of FIG. 1b target, combining the two prior to or by way of graphics adapter 27, as well as receiving and processing data from pointing device 10 to determine the pointed-to location at the may be identical to that in the presentation example of FIG. 1a; indeed, embodiments of this invention allow the same speaker SPKR may interact with the same presentation in 40 displayed image. Alternatively, it is contemplated that sepa front of the same audience both from a distance as shown in FIG. 1a, and at display 20 as shown in FIG. 1b. In either case, as described in further detail in the above rate functional systems external to computer 22 may carry out one or more of the functions of target generator 23, receiver 24, and positioning circuitry 25, such that computer 22 can be realized as a conventional computer operating without modi incorporated U.S. Pat. No. 8,217,997 and below in this description in connection with particular embodiments of the 45 ?cation; in this event, graphics adapter 27 could itself consti tute an external function (or be combined with one or more of invention, speaker SPKR carries out this interaction by way of pointing device 10, which is capable of capturing all or part the other functions of target generator 23, receiver 24, and of the image at display 20 and of interacting with a pointed-to positioning circuitry 25, external to computer 22), or altema tively be realized within computer 22, to which output from (or aimed-at) target location at that image. Pointing device 10 in the examples of FIGS. 1a and 1b wirelessly communicates 50 target generator 23 is presented. Other various alternative this pointed-to location at display 20 and other user com implementations of these functions are also contemplated. In any event, it is contemplated that computer 22, positioning circuitry 25, target generator 23, and other functions involved in the generation of the images and positioning targets dis invention, remote interactivity with computer 22 is carried out. 55 played at graphics display 20, will include the appropriate mands from speaker SPKR, to receiver 24 and thus to com puter 22. In this manner, according to embodiments of this Referring to FIG. 2a, a generalized example of the con struction of an interactive display system useful in environ ments such as those shown in FIGS. 1a and 1b, according to embodiments of this invention, will now be described. As program memory in the form of computer-readable media storing computer program instructions that, when executed by its processing circuitry, will carry out the various functions and operations of embodiments of the invention as described shown in FIG. 2a, this interactive display system includes 60 in this speci?cation. It is contemplated that those skilled in the art having reference to this speci?cation will be readily able to pointing device 10, projector 21, and display screen 20. In this arrange the appropriate computer hardware and correspond embodiment of the invention, computer 22 includes the appropriate functionality for generating the “payload” ing computer programs for implementation of these embodi images to be displayed at display screen 20 by projector 21, ments of the invention, without undue experimentation. such payload images intended for viewing by the audience. 65 The content of these payload images is interactively con trolled by a human user via pointing device 10, according to Pointing device 10 in this example includes a camera func tion consisting of optical system 12 and image sensor 14. With pointing device 10 aimed at display 20, image sensor 14 US 8,982,050 B2 7 8 is exposed with the captured image, which corresponds to all munications with one another, in which case data correspond or part of image I at display 20, depending on the distance ing to the size, shape, andposition of the positioning targets as between pointing device 10 and display 20, the focal length of lenses within optical system 12, and the like. Image capture subsystem 16 includes the appropriate circuitry known in the art for acquiring and storing a digital representation of the captured image at a particular point in time selected by the displayed at display 20 can be transmitted to pointing device user, or as captured at each of a sequence of sample times. Pointing device 10 also includes actuator 15, which is a con ventional push-button or other switch by way of which the user of pointing device 10 can provide user input in the nature of a mouse button, to actuate an image capture, or for other functions as will be described below and as will be apparent to those skilled in the art. In this example, one or more inertial sensors 17 are also included within pointing device 10, to assist or enhance user interaction with the displayed content; 10' for comparison. In either case, positioning circuitry 25, 25' (hereinafter referred to generically as positioning circuitry 25) determines the location at display 20 at which pointing device 10, 10' (hereinafter referred to generically as pointing device 10) is aimed, as will be described in detail below. As described in the above-incorporated U.S. Pat. No. 8,217,997, positioning cir cuitry 25 performs “absolute” positioning, in the sense that the pointed-to location at the display is determined with ref erence to a particular pixel position within the displayed image. As described in Us. Pat. No. 8,217,997, image cap ture sub system 16 captures images from two or more frames, those images including one or more positioning targets that are presented as patterned modulation of the intensity (e.g., variation in pixel intensity) in one display frame of the visual payload, followed by the same pattern but with the opposite sors. 20 modulation in a later (e.g., the next successive) frame. FIGS. 3a through 3d illustrate an example of absolute In this example of FIG. 2a, pointing device 10 is operable positioning as carried out by positioning circuitry 25, for the to forward, to positioning circuitry 25, signals that corre examples of such inertial sensors include accelerometers, magnetic sensors (i.e., for sensing orientation relative to the earth’s magnetic ?eld), gyroscopes, and other inertial sen spond to the captured image acquired by image capture sub case in which pointing device 10 is not in motion. FIG. 3a system 16. This communications function is performed by illustrates an image generated by computer 22 and target wireless transmitter 18 in pointing device 10, along with its 25 generator 23, displayed via graphics adapter 27 at display 20, and captured by the ?eld of view of image capture subsystem internal antenna A, by way of which radio frequency signals 16 in frame k. In this example, the image contains positioning (e. g., according to a conventional standard such as Bluetooth or the appropriate IEEE 802.11 standard) are transmitted. target 60(k) and visible element 62(k), each of which is darker Transmitter 18 is contemplated to be of conventional con than the background in the captured image. FIG. 3b illustrates struction and operation for encoding, modulating, and trans 30 the image generated and displayed in the next displayed mitting the captured image data, along with other user input frame k+l, and captured by image capture subsystem 16, that and control signals via the applicable wireless protocol. In image including positioning target 60(k) and visible element this example, receiver 24 is capable of receiving the transmit 62(k). Because pointing device 10 has not moved between the ted signals from pointing device 10 via its antenna A, and of times of frames k, k+l, positioning targets 60(k), 60(k+l) are demodulating, decoding, ?ltering, and otherwise processing 35 in the same position in the images of both frames, as are the received signals into a baseband form suitable for pro cessing by positioning circuitry 25. It is contemplated that the particular location of positioning circuitry 25 in the interactive display system of embodiments visible elements 62(k), 62(k+l) relative to one another. As described in the above-incorporated U.S. Pat. No. 8,217,997, positioning target 60(k+ l) is brighter than the background in frame k+l by an amount that is about the same as positioning of this invention may vary from system to system. It is not 40 target 60(k) is darker than the background in frame k. Visible particularly important, in the general sense, which hardware sub system (i.e., the computer driving the display, the pointing elements 62(k), 62(k+l) are darker than the background in both frames, as shown. FIG. 30 illustrates the captured image portions of FIGS. 3a and 3b averaged with one another, for example as is naturally combination thereof) performs the determination of the pointed-to location at display 20. In the example shown in 45 done by humans viewing the sequence of images at display FIG. 2a, as described above, positioning circuitry 25 is 20. Because positioning targets 60(k), 60(k+ l) are in the same location for both frames k, k+ l , but modulate the background deployed in combination with computer 22 and target gen device, a separate subsystem in the video data path, or some erator function 23, in a system that combines the functions of in a manner complementary to one another, the average image generating the displayed images I and of determining the AVG60 of the two positioning targets 60(k), 60(k+l) is simply location at the displayed images I at which pointing device 10 50 the background level of the displayed image, with no visible modulation (as suggested in FIG. 30 by the dashed lines at into the same element of the system. location AVG60). On the other hand, the average frame AVG FIG. 2b illustrates an alternative generalized arrangement (k, k+l) of frames k, k+l results in darker element AVG62 of an interactive display system according to embodiments of relative to the background. Element 62 is thus visible, and this invention. This system includes projector 21 and display 55 positioning target 60 is invisible, to humans in the audience of 20 as in the example of FIG. 2b, with projector 21 projecting display 20, as intended. According to the approach described in Us. Pat. No. payload image content and positioning target image content generated by computer 22 as described above. In this 8,217,997, positioning circuitry 25 subtracts the captured image data from these two frames k, k+l from one another. example, pointing device 10' performs some or all of the computations involved in determining the location at display 60 FIG. 3d illustrates difference frame A(k, k+l) resulting from 20 at which it is currently pointing. As such, in addition to a such subtraction; as evident in FIG. 3d, the (constant) back ground image cancels out and is not visible. Similarly, ele camera (lens 12, image sensor 14, and image capture 16), positioning device 10' includes positioning circuitry 25', ments 62(k), 62(k+l) cancel each other out in difference along with wireless transmitter 18. Conversely, computer 22 frame A(k, k+l), and as such element 62 is invisible for is coupled to receiver 24, as before. Alternatively, transmitter 65 purposes of the positioning task, as indicated by difference 18 and receiver 24 may be each be implemented as transceiv element 062 of FIG. 3d. On the other hand, because position ers, capable of both receiving and transmitting wireless com ing targets 60(k), 60(k+l) modulate the background in a man is aimed (and decoding the commands associated therewith) US 8,982,050 B2 10 ner complementary to one another, at the same location of the captured image for both of frames k, k+1, subtraction of the captured image of frame k+1 from that of frame k serves to reinforce the modulation, enabling the recovery of machine visible positioning target pattern A60 as shown in FIG. 3d. As described in Us. Pat. No. 8,217,997, positioning circuitry 25 is then able to determine location, size, and orientation from the location of positioning target pattern A60 in difference frame A(k, k+1), and thus identify the aiming point of point ing device 10 relative to display 20. As evident from FIG. 3a through 3d and the above descrip tion, positioning circuitry 25 operates quite well in identify ing human-invisible positioning targets 60 in the case in which pointing device 10 does not move signi?cantly from the time of one frame to the time of the next. However, rapid inter-frame motion of pointing device 10 between frames will move the location of positioning target 60 within the captured image portion from one frame to the next. This movement has been observed to create dif?culties in the absolute positioning 062 does not fully overlay visible elements 62(j), 62(j+1), leaving vestiges that remain visible to positioning circuitry 25 in difference frame AU, j+1). These vestigial visible elements 62(j), 62(j+1) can complicate the absolute positioning deter mination, particularly if those vestiges somewhat resemble a feature of positioning target 60, or overlap and thus distort positioning target 60, in either case reducing the likelihood that positioning circuitry 25 recognizes a positioning target pattern and calculates an accurate position. This issue is exacerbated by greater inter-frame motion, as shown in FIG. 4d. In this example, a more rapid movement of pointing device 10 between frames j, j+1 amounts to a shift shown by vector 3A (in the same direction but of three times the magnitude as vector A in FIGS. 4b and 40) between the l