public class KafkaSource extends AbstractPollableSource implements Configurable, BatchSizeSupported
kafka.consumer.group.id: the group ID of consumer group. Required
kafka.topics: the topic list separated by commas to consume messages from. Required
maxBatchSize: Maximum number of messages written to Channel in one batch. Default: 1000
maxBatchDurationMillis: Maximum number of milliseconds before a batch (of any size) will be written to a channel. Default: 1000
kafka.consumer.*: Any property starting with "kafka.consumer" will be passed to the kafka consumer So you can use any configuration supported by Kafka 0.9.0.X useFlumeEventFormat: Reads events from Kafka Topic as an Avro FlumeEvent. Used in conjunction with useFlumeEventFormat (Kafka Sink) or parseAsFlumeEvent (Kafka Channel)
| Modifier and Type | Class and Description |
|---|---|
class |
KafkaSource.Subscriber<T>
This class is a helper to subscribe for topics by using
different strategies
|
PollableSource.Status| Constructor and Description |
|---|
KafkaSource() |
| Modifier and Type | Method and Description |
|---|---|
protected void |
doConfigure(Context context)
We configure the source and generate properties for the Kafka Consumer
Kafka Consumer properties are generated as follows:
1.
|
protected PollableSource.Status |
doProcess() |
protected void |
doStart() |
protected void |
doStop() |
long |
getBatchSize()
Returns the batch size
|
configure, getBackOffSleepIncrement, getMaxBackOffSleepInterval, processgetChannelProcessor, getLifecycleState, getName, getStartException, isStarted, setChannelProcessor, setLifecycleState, setName, start, stop, toStringclone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, waitconfiguregetChannelProcessor, setChannelProcessorgetLifecycleState, start, stopgetName, setNamepublic long getBatchSize()
BatchSizeSupportedgetBatchSize in interface BatchSizeSupportedprotected PollableSource.Status doProcess() throws EventDeliveryException
doProcess in class AbstractPollableSourceEventDeliveryExceptionprotected void doConfigure(Context context) throws FlumeException
doConfigure in class BasicSourceSemanticscontext - FlumeExceptionprotected void doStart()
throws FlumeException
doStart in class BasicSourceSemanticsFlumeExceptionprotected void doStop()
throws FlumeException
doStop in class BasicSourceSemanticsFlumeExceptionCopyright © 2009-2022 Apache Software Foundation. All Rights Reserved.