public abstract class AbstractRpcSink extends AbstractSink implements Configurable, BatchSizeSupported
RpcClient.
The destination is an instance of Flume's .AvroSource or ThriftSource (based on
which implementation of this class is used), which
allows Flume agents to forward to other Flume agents, forming a tiered
collection infrastructure. Of course, nothing prevents one from using this
sink to speak to other custom built infrastructure that implements the same
RPC protocol.
Events are taken from the configured Channel in batches of the
configured batch-size. The batch size has no theoretical limits
although all events in the batch must fit in memory. Generally, larger
batches are far more efficient, but introduce a slight delay (measured in
millis) in delivery. The batch behavior is such that underruns (i.e. batches
smaller than the configured batch size) are possible. This is a compromise
made to maintain low latency of event delivery. If the channel returns a null
event, meaning it is empty, the batch is immediately sent, regardless of
size. Batch underruns are tracked in the metrics. Empty batches do not incur
an RPC roundtrip.
Configuration options
| Parameter | Description | Unit (data type) | Default |
|---|---|---|---|
| hostname | The hostname to which events should be sent. | Hostname or IP (String) | none (required) |
| port | The port to which events should be sent on hostname. | TCP port (int) | none (required) |
| batch-size | The maximum number of events to send per RPC. | events (int) | 100 |
| connect-timeout | Maximum time to wait for the first Avro handshake and RPC request | milliseconds (long) | 20000 |
| request-timeout | Maximum time to wait RPC requests after the first | milliseconds (long) | 20000 |
| compression-type | Select compression type. Default is "none" and the only compression type available is "deflate" | compression type | none |
| compression-level | In the case compression type is "deflate" this value can be between 0-9. 0 being no compression and 1-9 is compression. The higher the number the better the compression. 6 is the default. | compression level | 6 |
Metrics
TODO
Implementation Notes: Any implementation of this class must override the initializeRpcClient(Properties) method. This method will be called whenever this sink needs to create a new connection to the source.Sink.Status| Constructor and Description |
|---|
AbstractRpcSink() |
| Modifier and Type | Method and Description |
|---|---|
void |
configure(Context context)
Request the implementing class to (re)configure itself.
|
long |
getBatchSize()
Returns the batch size
|
protected abstract RpcClient |
initializeRpcClient(Properties props)
Returns a new RpcClient instance configured using the given
Properties object.
|
Sink.Status |
process()
Requests the sink to attempt to consume data from attached channel
|
void |
start()
The start() of RpcSink is more of an optimization that allows connection
to be created before the process() loop is started.
|
void |
stop()
Stops a service or component.
|
String |
toString() |
getChannel, getLifecycleState, getName, setChannel, setNamepublic void configure(Context context)
ConfigurableRequest the implementing class to (re)configure itself.
When configuration parameters are changed, they must be reflected by the component asap.
There are no thread safety guarantees on when configure might be called.
configure in interface Configurableprotected abstract RpcClient initializeRpcClient(Properties props)
props - public void start()
start in interface LifecycleAwarestart in class AbstractSinkpublic void stop()
LifecycleAwareStops a service or component.
Implementations should determine the result of any stop logic and effect
the return value of LifecycleAware.getLifecycleState() accordingly.
stop in interface LifecycleAwarestop in class AbstractSinkpublic String toString()
toString in class AbstractSinkpublic Sink.Status process() throws EventDeliveryException
SinkRequests the sink to attempt to consume data from attached channel
Note: This method should be consuming from the channel within the bounds of a Transaction. On successful delivery, the transaction should be committed, and on failure it should be rolled back.
process in interface SinkEventDeliveryException - In case of any kind of failure to
deliver data to the next hop destination.public long getBatchSize()
BatchSizeSupportedgetBatchSize in interface BatchSizeSupportedCopyright © 2009-2022 Apache Software Foundation. All Rights Reserved.